Commands for interrogating the log
l – look at the most recent items in the log
m – look at the log from the beginning of the day
y – look at yesterday’s log
t – tail the log dynamically
f – check the log for failures
r – resend files
b – block a destination
x – interrogate and resend from the woops queue
The FingerPost Information Processor (The FIP) is a message-processing system.
The Information Processor is not only a file-server and classical message-switch but also is able to present messages to the final user in a form best suited to that application so that post-processing is unnecessary.
A message can be either a text file in any form – 5,6,7 or 8 bit working – or a binary file such as a wire-photo or a typesetter file.
Message Processing and Distribution : Full character set conversion and string replacement. Automatic routing using header fields (for wire service files this usually means the original header too). This implies that different messages from the same source can be treated differently – tv lists for example can be formatted and typeset directly while tv text files are routed to the main editorial computer. Messages from important sources can be duplicated or run redundantly for security reasons.
System Control : including starting and stopping processes, queue control, an on-line help facility.
Intercept and Error reporting : Wrongly routed messages are intercepted for the operator to process while the error reporting can be used in tracking line and equipment faults and loss of service. A non-destructive line monitor shows activity on any incoming port.
Statistics, logging and accounting : Parameter files for checking : Traffic – what messages have passed to and from which destination. Access – who has read which mailbox. Failure tracking – list all failures. Customised list – all items of a particular type.
Rerouting and archival retrieval : Messages can be retrieved from the Database and routing to the original or a new destination. Files are saved in date-source order and can be selectively purged depending on both disk size and file importance. Off-line storage and file maintenance are via standard Unix commands.
In all cases throughout this manual
|%||standard Unix prompt|
|ip-wires:||FIP prompt, where wires = hostname|
commands to input are in italicsmachine responses are in roman
eg: % ip ip-wires: q %
where queues or directories are specified, they are usually specified as relative to your home directory: this is generally /fip.
• Linux (Ubuntu, Fedora and RedHat) kernels – 2.4, 2.6, 3.0
• Sun Solaris 9,10 – sparc and x86
• Windows 2003 / 2008
• IBM AIX 4 and 5
• Mac OSX 10.5++
We call them Unixboxes just so we show no favours…..
The program IP is used to control the FIP system. It allows you to start and stop other programs, review the log, see what is waiting etc.
It is important that IP is left running at all times as not only is it your interface into the FIP system, but it also runs a number of routine checks in the background: it checks for timeouts & failures and will restart programs it deems to have halted etc. IP will also check for certain failures and blockages on the Spiders, resetting the port where possible. This screen should be logged-on as teh user who started the IP programs in order to have the correct authority to change the system.
To start the IP program type IP at the Unix prompt:
% ip ip-wires : the prompt returned by IP.
IP is command-line driven and options should be entered at the prompt :
where wires is the localhost name of that system.
s : show status
The status command shows what processes are standard for this machine and what is running at the moment.
Three flavours are available :
||which gives just the name, host and on/off flags|
||which gives the display below|
||which adds the system PID (process ID) to the display below|
NAME GROUP PROGRAM ON extspt fip1 vwire -s spider1 -p 1102 -n extspt ON reuter fip1 wire -s spider1 -p 1103 -n reuter ON pa fip1 wire -s spider1 -p 1104 -n pa ON afp fip1 wire -s spider1 -p 1105 -n afp ON ap fip1 wire -s spider1 -p 1106 -n uns ON coi fip1 wire -s spider1 -p 1107 -n coi -x ON m7500 fip1 wire -s spider1 -p 1108 -n m7500 -x .. tlx1 fip1 telex -s spider1 -p 1109 -n 9419611 .. fax fip1 fax -s spider1 -p 1110 -n fax -q gnochi .. tph0521 fip1 ipbox -s spider1 -p 1111 -n tph0521 -t .. pss1_3 fip1 ipbox -s spider1 -p 1112 -n pss1_3 -x ON route local iproute ON post local ippost ON xchg local ipxchg ON wheel local ipwheel ON 2net local ip2net ON xnet local ipxnet ON redun local ipredun ON atexone local ipout ON atextwo local ipout ON atexthree local ipout
In the above example it is possible to have programs running against ports 2-12 on spider1. All local programs doing the internal processing and routing of copy are running (ie – the left hand column is ON), as are all incoming wire programs. Telex, Fax, PSS and a dial-up modem have all been disabled.
The Host column provides a means of grouping programs together by type: in this case all programs termed fip1 are incoming processes, all termed local are internal.
The program column describes the actual program running for that service/process, and where required the Spider Port that program is running against & any switches used.
IP gets the information on what to check from file SYSTEM in ~tables/sys. Once read it keeps a copy with all the pids for its own purposes.
The PID is the process ID or system number (see Unix command ‘ps’ to review).
Processes can be activated singularly or in groups. The syntax is :
ip-wires: a name
ip-wires: a program
ip-wires: a host
ip-wires: a all
Normally you will get a message :
* STARTED: name
to indicate that the program is running. You will always be notified if you try to activate an unknown service or group (ie if you make a typing mistake). If the process is already running, you are notified as such.
Using the previous example:
A single wire can be started with a (for activate) :
ip-wires: a pa
All processes to send copy to editorial systems can be started with :
ip-wires: a ipout
All processes running on a particular host can be started with:
ip-wires: a hostname
ie: in the previous example
ip-wires: a fip would activate all incoming wires
All processes in the SYSTEM file can be started with a :
ip-wires: a all
In the same way that processes can be activated singularly or in groups, they can be stopped or killed. The syntax is :
ip-wires: a name
ip-wires: k program
ip-wires: k host
ip-wires: k all
Normally you will get a message :
* STOPPED: name
If the process is already disabled, it will tell you.
A single incoming wire can be killed with a :
ip-wires: k reuter
All incoming wires running the wire program can be stopped with a :
ip-wires: k wire
All IP processes running in a group (usually a host) can be stopped with:
ip-wires: k local
All IP processes running on that machine can be stopped with:
ip-wires: k all
N.B. It is good practice to always check the status of the system using the
s all command after killing or activating any processes. It is always best to be fairly specific when killing processes: killing all wires when it is merely PA you want to stop is counterproductive; likewise killing all of the local processes when you merely want to stop traffic on to one Atex network can cause severe problems. Also: See redundancy later.
l, m, y, t & f : show, more, tail item log and check log for failures.
l – look at the item log
All incoming messages plus anything notable that happens to the system is logged in an item log (stored as log/all) and can thus be reviewed.
will give you a screenful of the last items through. It is also possible to get a screenful of a selection of items by specifying a search string. For example :
ip-wires: l REU
will look for the last items with REU in (remember this is a case-sensitive system).
Any character or character string that is in the log text file can be searched for – date, time, program name, destination, item etc. All items are logged by the time & date they come in, the program & service they came in on, and their designated destinations. There is also a flag preceded by an exclamation mark, which gives some information as to the type of item logged: so you can select what you look at in your log file by:
|ip-wires: l !s||will tell you the last few programs that have been started or stopped||ip-wires: l !x||log of failures (easier to do an f see below)||ip-wires: l !i||log of inbound wire traffic||ip-wires: l !o||log of outbound wire traffic||ip-wires: l !f||log of fax traffic||ip-wires: l !t||log of telex messages|
m – more/scroll the item log
The log can also be scrolled from the time it was last purged by :
|or||ip-wires: m REU||for a selection containing the string “REU”|
This will scroll you through the item log from the top. In general the log is purged nightly, towards midnight: this is because the log file can become very large and so cumbersome to move through.
Note You can also narrow down what you are viewing be typing “m a_string_in_the_log”. So
m ipedsys will display all lines in the log containing “ipedsys” since the last time it was purged.
y – more/scroll the previous day(s) item log
The log can also be scrolled from the time it was last purged by :
|ip-wires: y 1||page through yesterday’s log|
|or||ip-wires: y 3 REU||for a selection from 3 days ago|
The digit can be a number from 1 to 9 for 9 days ago. Generally however most sites leave only ther last 5 days worth online. If the log of that day does not exist, a message appears telling you so.
This will scroll you through the previous day(s) item log from the top. In general the log is purged nightly, towards midnight: this is because the log file can become very large and so cumbersome to move through.
Note, with this command you do not have the ability to select the service you are interested in as above; however you do have the simple search functionality provided by the Unix more command, and the ability to page through the file using a space bar to move forward a page at a time.
t – TAIL THE ITEM LOG
The log can left displaying traffic as items pass thru using the tail command.
To exit, type CntrlC. This returns you to the main IP prompt.
This uses the Unix tail command which will constantly update the display as traffic moves through the system.
Note that at midnight or when the item log is purge you will have to stop and restart this command as the file it is looking at is changed.
Any program failures or network problems can be searched for with this command:
It gives a display such as :
Mon Mar 25 15:50:51 wire (1405) !x : Socket Error 61
w : waiting in the queues
Occasionally you may want to check that no messages have been caught somewhere by using the W command to check :
|or||ip-wires: w all|
|or||ip-wires: w queuename|
On its own, a ‘w’ will give the total number of files per queue as given in the WAIT parameter file in tables/sys.
Looking at Queue : 2brouted: Number of files :0Looking at Queue : 2go: Number of files :2Looking at Queue : xchg: Number of files :1Looking at Queue : 2atex: Number of files :102
Any large number – like here for queue 2atex – means a program has not been activated or (if it is a transmission program) the destination may be down or the program has malfunctioned. If the latter is true, use the STATUS and LOG commands to see what the program should be and why it is not running.
To actually see what files are waiting, use either the w all command (to get the contents of ALL the queues specified in the wait table that was set up at installation) or specify an actual queuename if only one is required.
The specific queue can be any in the spool area. So, should you need to look at a queue that is not listed in the wait table (say a queue to a printer called /fip/spool/print, which you may not normally need to check on) a:
ip-wires: w print
would give you all the relevant information
c : review/change the editorial system tables
Note – this command is little used as there are few systems left like this !!
For those systems where FIP front ends a traditional editorial system such as an Atex 9000, IP uses a variable format program to deliver copy via either Ethernet or Serial communications to the editorial computer.
For systems where the editorial system resides on the Unixbox, this command has no bearing.
Depending on the amount of traffic, several ports may be used. Depending on the size of the newspaper or publishing company, several networks of computers can be addressed.
In an ideal world, files for a certain system are sent directly to it or to a companion machine; so all sports items should be sent directly to the sports machine and not via the foreign desk.
However if an editorial system is down for maintenance or is overloaded, you can the reroute files via one of its companions using the C command :
For example, if you have an editorial network called atex, use the following command
ip-wires: c atex
IP replies with the current routing (either the Spider Port or the Ethernet Port or Serial Port) for each system on that network :
** This is the current table atex2 1209 atex4 1409 atex6 1609 ; the above file is ATEX.246 ** The following tables exist : ATEX.246 ATEX.2 ATEX.24 ATEX.26 ATEX.46 ** New extention or return to keep existing :
The other tables available have different routings that have been setup for your installation. To retain the existing routing, hit Return. To change, type the number (without the dot) of the table you require. The extension for each table signifies which systems on that network you want to send files. In general your ideal table is the one which allows maximum distribution: in the above case this would be ATEX.246. However if system 2 is down for maintenance, or severely overloaded, you may want to reroute files to 4 & 6 temporarily.
To change the routing to systems 4 & 6 just type 46 at the prompt given above.
In cases where the system is overloaded, you will need to decide whether you want to send the files to alternative systems, so reducing the processing on the overloaded system, but increasing the bus traffic between the systems, or whether you want to stop all files going to that system at all (in which case you should use the block command). Such decisions depend on the time of day – can journalists do without those files for a few minutes in order to keep the system up & running, or is it imperative that those files reach the editorial systems under any circumstances?
For those sites with more than one network, each group of systems have there own tables. For example, if there are three networks, business, sport and editorial, then each of the three will have separate groups of tables : eg :
The NETONE network will have files starting with NETONE. Likewise for NETTWO, NETTHREE etc. So: to change routings on NETONE, you would need to type –
ip-wires: c NETONE
For those sites running redundant FIP systems you may well have wires going into one half of an Atex pair from one Unixbox, and the other half from the other Unixbox: if you need to keep wire traffic completely off an Atex pair, do not forget that you will need to change the tables on both Unixboxes.
Do note that if one or more Atex systems are down for a long time, and there is no need to send wires to alternative systems (say for overnight maintenance) you may be better off holding the wires back on the Unixbox, rather than building up down queues on the remaining systems on that network. This is done using either the block command or by killing all wires to that network.
You cannot reroute files between networks, only between systems on any one network.
The changes made are resident until you reset them – if you route all files for a network to one system, do not forget to route them back to their ideal systems when maintenance/problems are done with. There is no automatic reset!
As a result, if you are having problems with a particular editorial system, it is worth checking that the change tables on the Unixbox have not been left so that all files are directed at that system, so causing it to fall over during busy periods.
b – block output to specific destinations
The Block command allows you to temporarily retain files for certain destinations on the Unixbox; this can be useful if a system is overloaded or down, or if you want to retain output copy for a particular destination for debugging purposes.
If an editorial system is down or overloaded, it may be that that system is taking files which although generally stored on that system are not necessarily needed on that system at present. For instance on Saturday afternoons, it may be essential that Sports traffic reach the editorial system, but there be no need for the business and city traffic at that very moment. In such a case you could block all traffic destined for business and city queues, and unblock them after the afternoon’s results are in.
To block traffic:
ip-wires: b current blocked destinations are: reufin do you want to Block (B) Unblock (U) or continue (return)? :
In the above example you can see that all files destined for reufin are blocked: to unblock them, or allow them over to the editorial system type a U ; you will then be prompted for which destination you want to unblock – you would then type reufin.
Should you want to block additional destinations you would type a B at the above prompt, and again a destination would be requested.
To see how many files are blocked at any one time use the w command:
either a w on its own, if block is set up to be in your wait table, or
ip-wires: w block
If you seem to be missing files for a specific destination on your editorial system, and a log on the Unixbox shows those files to have come in, it is always worth checking whether that destination is blocked on the Unixbox.
p : peek – monitor an incoming service
Peek monitors an incoming service without destroying the text.
It can also be run from the Unix prompt:
It is used to check on a service which has been down or is thought to be garbling as it gives information on the last file through and the last 900 characters that were received irrespective of whether they were correctly formatted or not.
It can be used with any incoming service : wires, telexes, faxes and mailboxes.
Peek prompts for the following information before starting :
|Service||This is the name of the service as defined in
the SYSTEM file in the ‘-n’ field of the
Port and device
If a service is under test and has not
been added to the SYSTEM file you can set the
actual port here. If the device is a TTY rather
than a SpiderPort, use the Port number defined
in the PEEK file in ~/tables/sys.
Refresh (10) Normally Peek will scan the incoming
every 10 secs. If you need to view more or less
often, adjust the refresh rate.
No of bits (7) defines the character length which
defaults to 7. If 5 is chosen, Peek will
automatically translate the characters from
baudot to ascii before displaying.Control Chrs (NO)Peek will display Line
Feeds/Carriage Returns as UnderScores and other
Control Chrs as ‘^’ unless this switch is YES. If you do
type a Yes here it will display the actual control characters
in full: again useful for debugging a new service.
To quit at any prompt, type (escape) (return). To return to the menu, type Control C.
Peek will scan the buffer and display until stopped with a control C.
r : review and/or resend items
All incoming messages are saved in daily archive files. Any item or group of items can be resent from these archives using :
Firstly we need to describe how the archive files work. All incoming messages – wire traffic, telexes, stringer files etc – are archived when passing through the IPWHEEL stage. For each 24 hour period, IPWHEEL will store incoming text in time order in the file for that service. At midnight, the old file is closed and a new one created. IP uses the julian date of the year to describe what day the archive refers to. Generally archive files are NOT deleted for several days. At any time you can either review and/or resend a single or a selection of messages from an archive file.
Usually each discrete incoming (or outgoing if IPBDCAST is used) wire service has a separate archive file but the system administrator may wish to combine several inputs in a single archive file. If there are several telexes for example, it is simpler to scan a single telex archive file rather than having to look in several.
In addition to incoming traffic, messages can be archived at two other stages :
- after the XCHG process. This is more appropiate for those messages in non-ascii character sets where the incoming text is not meaningful to normal humans whereas the XCHGed file is.
- on output to the editorial system, wire service or a foreign system. This is where an actual output log is required for some ulterior purpose. All archive files are kept in log/data.To resend or review type :
FIP replies :
-------- Resend ------- Archives Files for : Sat December 23 AFP AP PA REUTER Current archive is : * : Sat Dec 23 Select and send : S Read the archive : R Watch incoming files : W Search for a string : G Look at another day : D Page through USERS : U List archives again : L Quit : Q :
Type the one letter command (return) to do that option.
Read which Service : You can (return) here to use the current archive name.
It will then ‘tail’ the archive. Any new file being appended will be automatically displayed.
Quit exits resend. List will relist all the archives available for that day. Users will page through the USERS file of destinations Watch will prompt the name of the archive (case INsensitive) : Read will prompt the name of the archive (case INsensitive) :
Read which Service :
You can (return) here to use the current archive name.
It then prompts :
Normal Read : N or (return) Translate CR to NL : T Octal dump : O Return to Top Menu : Q
The 2 abnormal options are for files with no LineFeeds (only Carriage Returns) and binary files respectively. READ then ‘mores’ the file.
|Search String||will prompt for the string to search for|
Lookwill prompt how many days ago. Type the number of days you wish to go back.If, by mistake, you are now before the day you require, type a negative number to go forward.
Send will prompt the name of the archive (case INsensitive) :
Read which Service :
You can (return) here to use the current archive name.
It then prompts :
Send single file : F Send a sequence of files : S Resend a Bdcast sequence : B Send files for a dest'n : D Send all files : X List valid destinations : L Return to Top Menu : Q
Type the one letter code (return) to choose.
|Send a single file||type the filename (case INsensitive).|
Send a sequencetype the names of the first and last.
Resend a Bdcastused ONLY for the News Agency system – again type the first and last sequences.
Send files for a destination type that destination and every file that has been to that destination – for that day – will be resent.
List valid destinations will review/more the USERS file
For all sends, you are prompted for a time range :
Starting from time hh:mm :
put in a start time if necessary (defaults to the start of the archive at midnight).
Ending at time (hh:mm) : 23:59 :
put in an end time if necessary (defaults to the end of the archive at the next midnight OR the current time).
Resend to same destination (return) or new dest name :
Hit return to resend to the same as before or type in the new destination which must be in the USERS file or the resend will end up in the Intercept (woops) queue.
At any point in the above dialogue you can stop by hitting the ESCAPE key to quit.
If any files are found, the message appears :
** Resend this message : Yes No All Count Quit y/n/a/c/q :
…and a filename is given.
|type ‘y’||for that one to be sent|
type ‘n’for that file not to be sent
type ‘a’for this and all other files to be resent
type ‘c’if you merely want to count how many items there are in this selection.
It is often a good idea to use the count facility before sending files over to an editorial system: if you are likely to flood the network because there are a very large number of files, you may need to narrow your selection.
Reading or reviewing the log file is useful if there appear to be garbles on your editorial system: worth checking whether or not the files came in garbled before telephoning the agency. Remember that running a peek also helps to assess whether or not copy is garbled on receipt. It is also useful if there is a need to scan the days input for files on a certain subject – you can use the more search facility to identify files containing certain strings and then send them on to the editorial system.
Resend is particularly useful if files are missing on the editorial system for one reason or another: the system may have been down, and all files between a certain timeslot may be missing; a port may have been causing data corruption in which case the files can be sent again. See later note on problem with duplicates and CCM if you intend to send over files that already exist on an Atex system.
Resend can also be useful for old files: in general wire copy, telexes and stringer copy are kept far longer on the Unixbox than is possible on an Atex network. If copy has been purged (or spiked mistakenly), it can be retrieved from the archive file on FIP.
x : look at the intercept queue
The intercept queue or the woops queue is where all misdirected or unfortunately routed message end up:there tend to be few files that do end up in woops – usually when someone has been adding services and forgetting to direct the files to their proper destination, or if files come in from agencies with garbled header material, so the system cannot work out where they should go.
Normally the ‘W’ command will show you if there is any intercept traffic. Then an ‘X’ will let you process them :
ip-wires: x IP replies : **Check the woops queue ** Incomplete destination(filename) ** Reroute, Spike, Ignore, read Text, ?, read USERS or Quit : type 'r' to reroute the copy to another, but valid, destination. type 's' to spike or delete the file. type 'i' to ignore this message for now but leave it in the woops queue for later processing. type '?' to read the help file on woops. type 'u' to read the USERS file of valid destinations. type 't' to read the text of the message using the Unix command MORE; after which reprompt. type 'q' to quit woops and return to IP. As you can see you have the option to read the file (and so to see if it is important copy, garbled copy etc), spike it, reroute it - in which case you will be prompted for either another destination or the same destination again, ignore it or quit.
? or H : HELP
Various on-line help files are available both from FingerPost and, as they are ordinary text files, you own in-house team.
ip-wires: h to call the list of IP HELP file or ip-wires: h cmd to read the list of commands ip-wires: h peek gives some documentation on peek. ip-wires: h spider tells you how to reset a spider port or ip-wires: h yourfiles to read a user-written help files.
Typing an h will give you a list of help files available; you can then type an h followed by the appropriate file name for more information.
Any added help files should be places in help.
d : date
D just tells you then system date and time
j : julian date
J tells you the Julian date today or if specified with a number, the julian date so many days in the future or in the past :
ip-wires: j -4 ** Julian Date of : Tue Jul 18 : 197
This is useful for the archive logs and other sundry internals.
! : shell escape
There are some occassions when you do not wish to quit IP but you need to run another program at the same time. This is particularly relevant if you are working from a single terminal screen or a pc and so do not have X/Sunwindows available.
Use ‘!’ to escape back to the shell for a single command :
ip-wires: !ls /usr/dave ip-wires: !mail
!! : repeat last command
This will repeat the last command typed at the ip-wires: prompt
v : version number
The ‘V’ command will tell you which version of IP you are running. Most IP programs have this facility.
ip-wires: v ip : version : 0.15
for other ip programs use the -v switch :
ip-wires: ipout -v ipout : version : 1.14
q : quit
‘Q’ is for QUIT to leave the program. Note that normal Control C does not have much effect on IP, so that when in other programs such as MORE you can quickly escape if need be.
Notes and Handy Tips
Here we try to guide you through tracking of problems and possible solutions.
1. File Duplication on the Editorial System
In general you should not receive any duplicates.
Some file duplication may occur if you resend files, you have a flaky network or unixbox and the redundancy module is running or if parameter files for routing have been specified wrongly.
Some agencies roll through more than 999 articles a day and reset their sequence numbers accordingly – Reuters for example. For get over this problem you can send this traffic through a different tables/out/FORMAT for ipout. This can be used to replace the agency sequence number with a system generated one. Remember to save the original name/number somewhere in the header.
2. Files are not being received by an editorial network
A sort of checklist !
With any problem, first define what is wrong ! The symptom may be different from the cause.
On FIP, for all problems, always do :
f – to check failures
w – to check waitings
l – to check what has just passed
If IPMON and/or IPSCAN are running, look at what they say.
Are any queues blocked; have any programs reported errors; are there any other problems that you are unaware of ?
2.1 Are any of your networks receiving files? (for multi-network sites)
If not kill & activate ipout for all networks.
2.2 Is it only one network that is missing files
If not, kill & activate ipout for that network.
2.3 Is it all services via a certain Terminal Server/Spider?
If so Check that this is powered on? Is its activity light flickering? Power it off & on. Go through the process of resetting the Spider described later.
2.4 Are all files for a certain system not being received?
Are files are being received on the rest of the network?
Check that your reception program is running on the Editorial System.