Fip Users Guide

Introduction
ip: The main Fip management programme
s – check the system status
k & a – stopping (killing) and starting
(activating) processes

Commands for interrogating the log
l – look at the most recent items in the log
m – look at the log from the beginning of the day
y – look at yesterday’s log
t – tail the log dynamically
f – check the log for failures
other commands
r – resend files
b – block a destination
x – interrogate and resend from the woops queue

Tips and tricks


Introduction

The FingerPost Information Processor (The FIP) is a message-processing system.

The Information Processor is not only a file-server and classical message-switch but also is able to present messages to the final user in a form best suited to that application so that post-processing is unnecessary.

A message can be either a text file in any form – 5,6,7 or 8 bit working – or a binary file such as a wire-photo or a typesetter file.

Message Processing and Distribution : Full character set conversion and string replacement. Automatic routing using header fields (for wire service files this usually means the original header too). This implies that different messages from the same source can be treated differently – tv lists for example can be formatted and typeset directly while tv text files are routed to the main editorial computer. Messages from important sources can be duplicated or run redundantly for security reasons.

System Control : including starting and stopping processes, queue control, an on-line help facility.

Intercept and Error reporting : Wrongly routed messages are intercepted for the operator to process while the error reporting can be used in tracking line and equipment faults and loss of service. A non-destructive line monitor shows activity on any incoming port.

Statistics, logging and accounting : Parameter files for checking : Traffic – what messages have passed to and from which destination. Access – who has read which mailbox. Failure tracking – list all failures. Customised list – all items of a particular type.

Rerouting and archival retrieval : Messages can be retrieved from the Database and routing to the original or a new destination. Files are saved in date-source order and can be selectively purged depending on both disk size and file importance. Off-line storage and file maintenance are via standard Unix commands.

In all cases throughout this manual

% standard Unix prompt
ip-wires: FIP prompt, where wires = hostname

commands to input are in italicsmachine responses are in roman

eg:	% ip
	ip-wires: q
	%

where queues or directories are specified, they are usually specified as relative to your home directory: this is generally /fip.

• Linux (Ubuntu, Fedora and RedHat) kernels – 2.4, 2.6, 3.0
• Sun Solaris 9,10 – sparc and x86
• Windows 2003 / 2008
• IBM AIX 4 and 5
• Mac OSX 10.5++

We call them Unixboxes just so we show no favours…..


The main FIP management program : IP

The program IP is used to control the FIP system. It allows you to start and stop other programs, review the log, see what is waiting etc.

It is important that IP is left running at all times as not only is it your interface into the FIP system, but it also runs a number of routine checks in the background: it checks for timeouts & failures and will restart programs it deems to have halted etc. IP will also check for certain failures and blockages on the Spiders, resetting the port where possible. This screen should be logged-on as teh user who started the IP programs in order to have the correct authority to change the system.

To start the IP program type IP at the Unix prompt:

	% ip
	ip-wires :		the prompt returned by IP.

IP is command-line driven and options should be entered at the prompt :

	ip-wires :

where wires is the localhost name of that system.

s : show status

The status command shows what processes are standard for this machine and what is running at the moment.

Three flavours are available :

ip-wires: s which gives just the name, host and on/off flags
ip-wires: s all which gives the display below
ip-wires: s pid which adds the system PID (process ID) to the display below

System Example

NAME           GROUP    PROGRAM
ON extspt	fip1	vwire -s spider1 -p 1102 -n extspt
ON reuter	fip1	wire  -s spider1 -p 1103 -n reuter
ON pa		fip1  	wire  -s spider1 -p 1104 -n pa
ON afp		fip1   	wire  -s spider1 -p 1105 -n afp
ON ap		fip1   	wire  -s spider1 -p 1106 -n uns
ON coi		fip1   	wire  -s spider1 -p 1107 -n coi   -x
ON m7500	fip1   	wire  -s spider1 -p 1108 -n m7500 -x
.. tlx1		fip1   	telex -s spider1 -p 1109 -n 9419611
.. fax		fip1   	fax  -s spider1 -p  1110 -n fax -q gnochi
.. tph0521	fip1   	ipbox -s spider1 -p 1111 -n tph0521 -t
.. pss1_3	fip1   	ipbox -s spider1 -p 1112 -n pss1_3  -x
ON route	local	iproute
ON post		local	ippost
ON xchg	local	ipxchg
ON wheel	local	ipwheel
ON 2net		local	ip2net
ON xnet	local	ipxnet
ON redun	local	ipredun
ON atexone	local	ipout
ON atextwo	local	ipout
ON atexthree	local	ipout

In the above example it is possible to have programs running against ports 2-12 on spider1. All local programs doing the internal processing and routing of copy are running (ie – the left hand column is ON), as are all incoming wire programs. Telex, Fax, PSS and a dial-up modem have all been disabled.

The Host column provides a means of grouping programs together by type: in this case all programs termed fip1 are incoming processes, all termed local are internal.

The program column describes the actual program running for that service/process, and where required the Spider Port that program is running against & any switches used.

IP gets the information on what to check from file SYSTEM in ~tables/sys. Once read it keeps a copy with all the pids for its own purposes.

The PID is the process ID or system number (see Unix command ‘ps’ to review).


k & a : killing and activating processes

Processes can be activated singularly or in groups. The syntax is :

* ip-wires: a name
* or ip-wires: a program
* or ip-wires: a host
* or ip-wires: a all

Normally you will get a message :

	* STARTED: name

to indicate that the program is running. You will always be notified if you try to activate an unknown service or group (ie if you make a typing mistake). If the process is already running, you are notified as such.

Using the previous example:

A single wire can be started with a (for activate) :

	ip-wires: a pa

All processes to send copy to editorial systems can be started with :

	ip-wires: a ipout

All processes running on a particular host can be started with:

	ip-wires: a hostname

ie: in the previous example

ip-wires: a fip		would activate all incoming wires

All processes in the SYSTEM file can be started with a :
ip-wires: a all

In the same way that processes can be activated singularly or in groups, they can be stopped or killed. The syntax is :

ip-wires: a name
* or ip-wires: k program
* or ip-wires: k host
* or ip-wires: k all

Normally you will get a message :

	* STOPPED: name

If the process is already disabled, it will tell you.

A single incoming wire can be killed with a :

	ip-wires: k reuter

All incoming wires running the wire program can be stopped with a :

	ip-wires: k wire

All IP processes running in a group (usually a host) can be stopped with:

	ip-wires: k local

All IP processes running on that machine can be stopped with:
ip-wires: k all

N.B. It is good practice to always check the status of the system using the s all command after killing or activating any processes. It is always best to be fairly specific when killing processes: killing all wires when it is merely PA you want to stop is counterproductive; likewise killing all of the local processes when you merely want to stop traffic on to one Atex network can cause severe problems. Also: See redundancy later.

l, m, y, t & f : show, more, tail item log and check log for failures.

l – look at the item log

All incoming messages plus anything notable that happens to the system is logged in an item log (stored as log/all) and can thus be reviewed.

	ip-wires: l

will give you a screenful of the last items through. It is also possible to get a screenful of a selection of items by specifying a search string. For example :

	ip-wires: l REU

will look for the last items with REU in (remember this is a case-sensitive system).

Any character or character string that is in the log text file can be searched for – date, time, program name, destination, item etc. All items are logged by the time & date they come in, the program & service they came in on, and their designated destinations. There is also a flag preceded by an exclamation mark, which gives some information as to the type of item logged: so you can select what you look at in your log file by:

ip-wires: l !s will tell you the last few programs that have been started or stopped ip-wires: l !x log of failures (easier to do an f see below) ip-wires: l !i log of inbound wire traffic ip-wires: l !o log of outbound wire traffic ip-wires: l !f log of fax traffic ip-wires: l !t log of telex messages


m – more/scroll the item log


The log can also be scrolled from the time it was last purged by :

ip-wires: m
or ip-wires: m REU for a selection containing the string “REU”

This will scroll you through the item log from the top. In general the log is purged nightly, towards midnight: this is because the log file can become very large and so cumbersome to move through.

Note You can also narrow down what you are viewing be typing “m a_string_in_the_log”. So m ipedsys will display all lines in the log containing “ipedsys” since the last time it was purged.


y – more/scroll the previous day(s) item log


The log can also be scrolled from the time it was last purged by :

ip-wires: y 1 page through yesterday’s log
or ip-wires: y 3 REU for a selection from 3 days ago

The digit can be a number from 1 to 9 for 9 days ago. Generally however most sites leave only ther last 5 days worth online. If the log of that day does not exist, a message appears telling you so.

This will scroll you through the previous day(s) item log from the top. In general the log is purged nightly, towards midnight: this is because the log file can become very large and so cumbersome to move through.

Note, with this command you do not have the ability to select the service you are interested in as above; however you do have the simple search functionality provided by the Unix more command, and the ability to page through the file using a space bar to move forward a page at a time.

t – TAIL THE ITEM LOG

The log can left displaying traffic as items pass thru using the tail command.

ip-wires: t

To exit, type CntrlC. This returns you to the main IP prompt.

This uses the Unix tail command which will constantly update the display as traffic moves through the system.

Note that at midnight or when the item log is purge you will have to stop and restart this command as the file it is looking at is changed.


f – checking for problems

Any program failures or network problems can be searched for with this command:

ip-wires: f

It gives a display such as :

Mon Mar 25 15:50:51	wire (1405) !x	: Socket Error 61

w : waiting in the queues

Occasionally you may want to check that no messages have been caught somewhere by using the W command to check :

ip-wires: w
or ip-wires: w all
or ip-wires: w queuename

On its own, a ‘w’ will give the total number of files per queue as given in the WAIT parameter file in tables/sys.
eg:

Looking at Queue : 2brouted: Number of files :0Looking at Queue : 2go: Number of files :2Looking at Queue : xchg: Number of files :1Looking at Queue : 2atex: Number of files :102

Any large number – like here for queue 2atex – means a program has not been activated or (if it is a transmission program) the destination may be down or the program has malfunctioned. If the latter is true, use the STATUS and LOG commands to see what the program should be and why it is not running.

To actually see what files are waiting, use either the w all command (to get the contents of ALL the queues specified in the wait table that was set up at installation) or specify an actual queuename if only one is required.

The specific queue can be any in the spool area. So, should you need to look at a queue that is not listed in the wait table (say a queue to a printer called /fip/spool/print, which you may not normally need to check on) a:

ip-wires: w print

would give you all the relevant information

c : review/change the editorial system tables

Note – this command is little used as there are few systems left like this !!

For those systems where FIP front ends a traditional editorial system such as an Atex 9000, IP uses a variable format program to deliver copy via either Ethernet or Serial communications to the editorial computer.

For systems where the editorial system resides on the Unixbox, this command has no bearing.

Depending on the amount of traffic, several ports may be used. Depending on the size of the newspaper or publishing company, several networks of computers can be addressed.

In an ideal world, files for a certain system are sent directly to it or to a companion machine; so all sports items should be sent directly to the sports machine and not via the foreign desk.

However if an editorial system is down for maintenance or is overloaded, you can the reroute files via one of its companions using the C command :

For example, if you have an editorial network called atex, use the following command

ip-wires: c atex

IP replies with the current routing (either the Spider Port or the Ethernet Port or Serial Port) for each system on that network :

** This is the current table
	atex2		1209
	atex4		1409
	atex6		1609
; the above file is ATEX.246
** The following tables exist :
	ATEX.246
	ATEX.2
	ATEX.24
	ATEX.26
	ATEX.46
** New extention or return to keep existing :

The other tables available have different routings that have been setup for your installation. To retain the existing routing, hit Return. To change, type the number (without the dot) of the table you require. The extension for each table signifies which systems on that network you want to send files. In general your ideal table is the one which allows maximum distribution: in the above case this would be ATEX.246. However if system 2 is down for maintenance, or severely overloaded, you may want to reroute files to 4 & 6 temporarily.

To change the routing to systems 4 & 6 just type 46 at the prompt given above.

In cases where the system is overloaded, you will need to decide whether you want to send the files to alternative systems, so reducing the processing on the overloaded system, but increasing the bus traffic between the systems, or whether you want to stop all files going to that system at all (in which case you should use the block command). Such decisions depend on the time of day – can journalists do without those files for a few minutes in order to keep the system up & running, or is it imperative that those files reach the editorial systems under any circumstances?

For those sites with more than one network, each group of systems have there own tables. For example, if there are three networks, business, sport and editorial, then each of the three will have separate groups of tables : eg :

The NETONE network will have files starting with NETONE. Likewise for NETTWO, NETTHREE etc. So: to change routings on NETONE, you would need to type –

ip-wires: c NETONE

For those sites running redundant FIP systems you may well have wires going into one half of an Atex pair from one Unixbox, and the other half from the other Unixbox: if you need to keep wire traffic completely off an Atex pair, do not forget that you will need to change the tables on both Unixboxes.

Do note that if one or more Atex systems are down for a long time, and there is no need to send wires to alternative systems (say for overnight maintenance) you may be better off holding the wires back on the Unixbox, rather than building up down queues on the remaining systems on that network. This is done using either the block command or by killing all wires to that network.

You cannot reroute files between networks, only between systems on any one network.

The changes made are resident until you reset them – if you route all files for a network to one system, do not forget to route them back to their ideal systems when maintenance/problems are done with. There is no automatic reset!

As a result, if you are having problems with a particular editorial system, it is worth checking that the change tables on the Unixbox have not been left so that all files are directed at that system, so causing it to fall over during busy periods.

b – block output to specific destinations

The Block command allows you to temporarily retain files for certain destinations on the Unixbox; this can be useful if a system is overloaded or down, or if you want to retain output copy for a particular destination for debugging purposes.

If an editorial system is down or overloaded, it may be that that system is taking files which although generally stored on that system are not necessarily needed on that system at present. For instance on Saturday afternoons, it may be essential that Sports traffic reach the editorial system, but there be no need for the business and city traffic at that very moment. In such a case you could block all traffic destined for business and city queues, and unblock them after the afternoon’s results are in.

To block traffic:

	ip-wires: b
	current blocked destinations are: reufin
	do you want to Block (B) Unblock (U) or continue (return)? :

In the above example you can see that all files destined for reufin are blocked: to unblock them, or allow them over to the editorial system type a U ; you will then be prompted for which destination you want to unblock – you would then type reufin.

Should you want to block additional destinations you would type a B at the above prompt, and again a destination would be requested.

To see how many files are blocked at any one time use the w command:
either a w on its own, if block is set up to be in your wait table, or

ip-wires: w block

If you seem to be missing files for a specific destination on your editorial system, and a log on the Unixbox shows those files to have come in, it is always worth checking whether that destination is blocked on the Unixbox.

p : peek – monitor an incoming service

Peek monitors an incoming service without destroying the text.

It can also be run from the Unix prompt:

	% peek

It is used to check on a service which has been down or is thought to be garbling as it gives information on the last file through and the last 900 characters that were received irrespective of whether they were correctly formatted or not.

It can be used with any incoming service : wires, telexes, faxes and mailboxes.

Peek prompts for the following information before starting :

Service This is the name of the service as defined in
the SYSTEM file in the ‘-n’ field of the
incoming service.

Port and device

If a service is under test and has not
been added to the SYSTEM file you can set the
actual port here. If the device is a TTY rather
than a SpiderPort, use the Port number defined
in the PEEK file in ~/tables/sys.

Refresh (10) Normally Peek will scan the incoming
every 10 secs. If you need to view more or less
often, adjust the refresh rate.

No of bits (7) defines the character length which
defaults to 7. If 5 is chosen, Peek will
automatically translate the characters from
baudot to ascii before displaying.Control Chrs (NO)Peek will display Line
Feeds/Carriage Returns as UnderScores and other
Control Chrs as ‘^’ unless this switch is YES. If you do
type a Yes here it will display the actual control characters
in full: again useful for debugging a new service.

To quit at any prompt, type (escape) (return). To return to the menu, type Control C.

Peek will scan the buffer and display until stopped with a control C.

r : review and/or resend items

All incoming messages are saved in daily archive files. Any item or group of items can be resent from these archives using :

	ip-wires: r

Firstly we need to describe how the archive files work. All incoming messages – wire traffic, telexes, stringer files etc – are archived when passing through the IPWHEEL stage. For each 24 hour period, IPWHEEL will store incoming text in time order in the file for that service. At midnight, the old file is closed and a new one created. IP uses the julian date of the year to describe what day the archive refers to. Generally archive files are NOT deleted for several days. At any time you can either review and/or resend a single or a selection of messages from an archive file.

Usually each discrete incoming (or outgoing if IPBDCAST is used) wire service has a separate archive file but the system administrator may wish to combine several inputs in a single archive file. If there are several telexes for example, it is simpler to scan a single telex archive file rather than having to look in several.

In addition to incoming traffic, messages can be archived at two other stages :

  • after the XCHG process. This is more appropiate for those messages in non-ascii character sets where the incoming text is not meaningful to normal humans whereas the XCHGed file is.
  • on output to the editorial system, wire service or a foreign system. This is where an actual output log is required for some ulterior purpose. All archive files are kept in log/data.To resend or review type :
    	ip-wires: r

    FIP replies :

    	-------- Resend -------
    	Archives Files for		: Sat December 23
    
    	AFP	AP	PA	REUTER
    
    	Current archive is	: *	: Sat Dec 23
    	Select and send		: S
    	Read the archive	: R
    	Watch incoming files	: W
    	Search for a string	: G
    	Look at another day	: D
    	Page through USERS 	: U
    	List archives again	: L
    	Quit			: Q :

    Type the one letter command (return) to do that option.

    Read which Service : You can (return) here to use the current archive name.
    It will then ‘tail’ the archive. Any new file being appended will be automatically displayed.

    Quit exits resend.
    List will relist all the archives available for that day.
    Users will page through the USERS file of destinations
    Watch will prompt the name of the archive (case INsensitive) :
    Read will prompt the name of the archive (case INsensitive) :

Read which Service :
You can (return) here to use the current archive name.
It then prompts :

			Normal Read		: N or (return)
			Translate CR to NL	: T
			Octal dump		: O
			Return to Top Menu	: Q

The 2 abnormal options are for files with no LineFeeds (only Carriage Returns) and binary files respectively. READ then ‘mores’ the file.

Search String will prompt for the string to search for

Lookwill prompt how many days ago. Type the number of days you wish to go back.If, by mistake, you are now before the day you require, type a negative number to go forward.

Send will prompt the name of the archive (case INsensitive) :

			Read which Service	:

You can (return) here to use the current archive name.
It then prompts :

			Send single file		: F
			Send a sequence of files	: S
			Resend a Bdcast sequence	: B
			Send files for a dest'n		: D
			Send all files			: X
			List valid destinations		: L
			Return to Top Menu		: Q

Type the one letter code (return) to choose.

Send a single file type the filename (case INsensitive).

Send a sequencetype the names of the first and last.

Resend a Bdcastused ONLY for the News Agency system – again type the first and last sequences.

Send files for a destination type that destination and every file that has been to that destination – for that day – will be resent.

List valid destinations will review/more the USERS file

For all sends, you are prompted for a time range :

		Starting from time hh:mm :

put in a start time if necessary (defaults to the start of the archive at midnight).

		Ending at time (hh:mm) : 23:59 :

put in an end time if necessary (defaults to the end of the archive at the next midnight OR the current time).

		Resend to same destination (return) or new dest name :

Hit return to resend to the same as before or type in the new destination which must be in the USERS file or the resend will end up in the Intercept (woops) queue.

At any point in the above dialogue you can stop by hitting the ESCAPE key to quit.

If any files are found, the message appears :

** Resend this message : Yes No All Count Quit y/n/a/c/q :

…and a filename is given.

type ‘y’ for that one to be sent

type ‘n’for that file not to be sent

type ‘a’for this and all other files to be resent

type ‘c’if you merely want to count how many items there are in this selection.

It is often a good idea to use the count facility before sending files over to an editorial system: if you are likely to flood the network because there are a very large number of files, you may need to narrow your selection.

Reading or reviewing the log file is useful if there appear to be garbles on your editorial system: worth checking whether or not the files came in garbled before telephoning the agency. Remember that running a peek also helps to assess whether or not copy is garbled on receipt. It is also useful if there is a need to scan the days input for files on a certain subject – you can use the more search facility to identify files containing certain strings and then send them on to the editorial system.

Resend is particularly useful if files are missing on the editorial system for one reason or another: the system may have been down, and all files between a certain timeslot may be missing; a port may have been causing data corruption in which case the files can be sent again. See later note on problem with duplicates and CCM if you intend to send over files that already exist on an Atex system.

Resend can also be useful for old files: in general wire copy, telexes and stringer copy are kept far longer on the Unixbox than is possible on an Atex network. If copy has been purged (or spiked mistakenly), it can be retrieved from the archive file on FIP.

x : look at the intercept queue

The intercept queue or the woops queue is where all misdirected or unfortunately routed message end up:there tend to be few files that do end up in woops – usually when someone has been adding services and forgetting to direct the files to their proper destination, or if files come in from agencies with garbled header material, so the system cannot work out where they should go.

Normally the ‘W’ command will show you if there is any intercept traffic. Then an ‘X’ will let you process them :

ip-wires: x

IP replies :
**Check the woops queue **
Incomplete destination(filename)
** Reroute, Spike, Ignore, read Text, ?, read USERS or Quit :

	type 'r' to reroute the copy to another, but valid,
		destination.

	type 's' to spike or delete the file.

	type 'i' to ignore this message for now but leave it in
		the woops queue for later processing.

	type '?' to read the help file on woops.

	type 'u' to read the USERS file of valid destinations.

	type 't' to read the text of the message using the 
		Unix command MORE; after which reprompt.

	type 'q' to quit woops and return to IP.

As you can see you have the option to read the file (and so to see if it is important copy, garbled copy etc), spike it, reroute it - in which case you will be prompted for either another destination or the same destination again, ignore it or quit.

? or H : HELP



Various on-line help files are available both from FingerPost and, as they are ordinary text files, you own in-house team.
	ip-wires: h			to call the list of IP HELP file
or	ip-wires: h cmd		to read the list of commands
	ip-wires: h peek		gives some documentation on peek.
	ip-wires: h spider		tells you how to reset a spider port
or	ip-wires: h yourfiles		to read a user-written help files.

Typing an h will give you a list of help files available; you can then type an h followed by the appropriate file name for more information.

Any added help files should be places in help.

d : date

D just tells you then system date and time

j : julian date

J tells you the Julian date today or if specified with a number, the julian date so many days in the future or in the past :

	ip-wires: j -4
	** Julian Date of : Tue Jul 18 : 197

This is useful for the archive logs and other sundry internals.

! : shell escape

There are some occassions when you do not wish to quit IP but you need to run another program at the same time. This is particularly relevant if you are working from a single terminal screen or a pc and so do not have X/Sunwindows available.

Use ‘!’ to escape back to the shell for a single command :
examples:

	ip-wires: !ls /usr/dave
	ip-wires: !mail

!! : repeat last command

This will repeat the last command typed at the ip-wires: prompt

v : version number

The ‘V’ command will tell you which version of IP you are running. Most IP programs have this facility.

For IP:

	ip-wires: v
	ip : version : 0.15

for other ip programs use the -v switch :

	ip-wires: ipout -v
	ipout : version : 1.14

q : quit

‘Q’ is for QUIT to leave the program. Note that normal Control C does not have much effect on IP, so that when in other programs such as MORE you can quickly escape if need be.


Notes and Handy Tips


Here we try to guide you through tracking of problems and possible solutions.

1. File Duplication on the Editorial System

In general you should not receive any duplicates.

Some file duplication may occur if you resend files, you have a flaky network or unixbox and the redundancy module is running or if parameter files for routing have been specified wrongly.

Some agencies roll through more than 999 articles a day and reset their sequence numbers accordingly – Reuters for example. For get over this problem you can send this traffic through a different tables/out/FORMAT for ipout. This can be used to replace the agency sequence number with a system generated one. Remember to save the original name/number somewhere in the header.

2. Files are not being received by an editorial network

A sort of checklist !

With any problem, first define what is wrong ! The symptom may be different from the cause.

On FIP, for all problems, always do :
f – to check failures
w – to check waitings
l – to check what has just passed

If IPMON and/or IPSCAN are running, look at what they say.

Are any queues blocked; have any programs reported errors; are there any other problems that you are unaware of ?

2.1 Are any of your networks receiving files? (for multi-network sites)
If not kill & activate ipout for all networks.

2.2 Is it only one network that is missing files
If not, kill & activate ipout for that network.

2.3 Is it all services via a certain Terminal Server/Spider?

If so Check that this is powered on? Is its activity light flickering? Power it off & on. Go through the process of resetting the Spider described later.

2.4 Are all files for a certain system not being received?

Are files are being received on the rest of the network?

Check that your reception program is running on the Editorial System.