Project: PHP-Nuke - 01. FileBase to Download Import - Analyse

AMBROSIA60-Portal  FileBase to Download Import Project



PHP-Nuke Download Section vs. FileBase
How to import from Filebase ...




Übersicht:
Start....Übersicht
Section 1....Analyse I, Filebase Synchronisation
Section 2....Umsetzung der Filebase Synchronisationsprozedur
Section 3....FileBase Import, Allgemeine Beschreibung
Section 4....Analyse II, Ticker Import
Section 5....Umsetzung des TIC Imports
Section 6....TIC Import, Allgemeine Beschreibung
Section 7....Einbindung DLIMP / DLTIC in die Maintenance
Section 8....Hinweis zur FileBase Security
Section 9....Übersicht und Status Gesamtprojekt
Section 10....References
Section 11....Download
Section 12....Weiterentwicklungen, Fixes


  • Section 1 - Analyse I, Filebase Synchronisation


[15.6.2005]

Download Section Update Prozess

Gegeben: AMBROSIA60 Filebase  Q:\dirs\subdirs     filebase files.bbs format
         BBS2WEB              Q:\BBS2WEB\www      HTM files

Erstellung:
         CATEGOR.LST          list of htm files
         CATEGORY.TXT         list of MySQL Nuke Download Categories
         DOWNLOAD.SQL         update list for import to MySql

categor.lst
  cd q:\bbs2web\www
  dir *.htm>cateor.lst

category.txt
  I:\Apache\mysql\bin>mysqldump --host=localhost --user=name --password=name
       --result-file=category.txt nuke nuke_downloads_categories

download.sql     x1)
  [new program]

Import to MySQL:
  I:\Apache\mysql\bin>mysql --host=localhost --user=name
       --password=name nuke<download.sql
x1) download.sql Format:
..............................................................................
insert into nuke_downloads_categories values (NULL, 'Box_Info',
 'AMBROSIA60 Box Infos','39');
insert into nuke_downloads_downloads values (NULL,'categorie#','0','Filename',
 'url x2)','description x3)','filedate yyyy-mm-dd hh:mm:ss x4)','Allfix','',
 '0','uli','0.0000','0','0','filesize x5)','version x6)','');
quit
..............................................................................
x2) URL: Link to filebase i.e.
http://ambrosia60.dd-dns.de/files/service/24000_fd/r24pnt.z40
Link aus *.htm files

x3) description from htm, 2nd line

x4) date + time from htm, fields Date: + Time:
    copy Date: yyyy-mm-dd + " " + Time: hh:mm:ss

x5) filesize from htm, field Size:

x6) version from htm, field description, 2nd line
    if day-### use ### with '#' and year-field -> i.e. day-161 -> #161/2005


Alternative?
 direkt aus FILES.BBS auslesen?
weiterhin noch unklar: URL Link
URL Link Base Path
  http://ambrosia60.dd-dns.de/files
ist klar, aber danach ... wie setzt sich
  /service/24000_fd/r24pnt.z40
zusammen?
BBS2WEB greift auf MAX FAREAS.DAT zu.
FAREAS.DAT setzt sich aus Definitionen aus FILEAREA.CTL zusammen,
das wiederum die Orginal Pfade enthaelt: i.e.  Q:\service\24000_fd

entweder [l:\max\]FILEAREA.CTL oder [q:\ra\]FLSEARCH.CTL auslesen
    basepath + path + filename = url-link

Auslesen FAREA.DAT ?  (seq. file)
offset  70   (ab pos 70 beginnt erste area definition)
areaname + chr(0) + path + chr(0) + accesslevel (klartext!) + chr(0)
 + description +    4x 16 bytes + 2 ... 

 chr(0) +  type datelist, type free ?!?
28 36 30 29 ³ 5F 42 6F 78 ³ 5F 49 6E 66 ³ 6F 00 40 00   (60)_Box_Info.@.
                                                ^^
00 00 29 00 ³ 00 00 17 00 ³ 01 00 0A 00 ³ 00 00 00 00   ................
1E 00 00 00 ³ 00 00 00 00 ³ 00 00 7A 00 ³ 00 00 40 03   ..........z...@.
                                                ^^
00 00 00 00 ³ 00 00 00 00 ³ 00 00 00 00 ³ 00 00 00 00   ................
00 00 00 00 ³ 00 00 00 00 ³ 00 00 00 00 ³ 00 00 00 46   ...............F
49 4C 45 4C ³ 49 53 54 00 ³ 51 3A 5C 66 ³ 69 6C 65 6C   ILELIST.Q:\filel
69 73 74 5C ³ 00 50 72 69 ³ 76 69 6C 00 ³ 46 69 6C 65   ist\.Privil.File
6C 69 73 74 ³ 65 6E 00 40 ³ 00 00 00 3F ³ 00 00 00 13   listen.@...?...#

00 01 00 05 ³ 00 00 00 00 ³ 00 1A 00 00 ³ 00 00 00 00   ................
00 00 00 69 ³ 00 00 00 40 ³ 03 00 00 00 ³ 00 00 00 00   ...i...@........
00 00 00 00 ³ 00 00 00 00 ³ 00 00 00 00 ³ 00 00 00 00   ................

00 00 00 00 ³ 00 00 00 00 ³ 57 49 32 00 ³ 51 3A 5C 44   ........WI2.Q:\D

3x16 + 8 + 8

last record:
057A5  20 20 00 40 ³ 00 00 00 4D ³ 00 00 00 1C ³ 00 01 00 08    .@...M........
             ^start
057B5  00 00 00 00 ³ 00 23 00 00 ³ 00 00 00 00 ³ 00 00 00 8D  .....#..........
057C5  00 00 00 40 ³ 03 00 00 00 ³ 00 00 00 00 ³ 00 00 00 00  ...@............
057D5  00 00 00 00 ³ 00 00 00 00 ³ 00 00 00 00 ³ 00 00 00 00  ................
057E5  00 00 00 00 ³ 54 52 4E 5F ³ 41 52 00 51 ³ 3A 5C 54 52  ....TRN_AR.Q:\TR
057F5  45 49 42 45 ³ 52 4E 5C 50 ³ 41 43 4B 45 ³ 52 5C 00 50  EIBERN\PACKER\.P
05805  72 69 76 69 ³ 6C 00 54 72 ³ 65 69 62 65 ³ 72 4E 65 74  rivil.TreiberNet
05815  5F 2D 5F 50 ³ 61 63 6B 70 ³ 72 6F 67 72 ³ 61 6D 6D 65  _-_Packprogramme
05825  20 20 20 20 ³ 20 20 20 20 ³ 20 20 20 20 ³ 20 20 20 00                 .
????

Yep, Offset faengt nicht bei 70 oder so an, sondern bei 0 ...
es sind jeweils komplette Records, mit binaerem VOrspann und
nachfolgendem PATH und DESCRIPTION. Laenge des Records scheint abhaengig
von der Pathlaenge und der Description ...


crawl thru files.bbs'ses
use files.bbs size as update indicator (filedate/time permanently changes!)
if filesize([path]files.bbs) <> stamp
  analyze files.bbs
  update sql script


flsearch.ctl alleine hilft nicht, da die struktur nicht
unbedingt mit der von fareas.dat uebereinstimmt -> use filearea.ctl
instead
weiterhin gruppierung und subcategories:
fidonet, webportale, filebase

fidonet  ..... use list
webportale ... use list
filebase ..... div13, div3, div27, service,point sonderfaelle
               ansonsten subcategory by root path
               ie. sds, pdn, gnf, gfd, win32  usw.


groups.lst

analyze:
readin groups and dirs
  exclude list  [EXCLUDE]
ansonsten, areagroup
[CATEGORIE]
or
[CATEGORIE\subcategorie]

and
[CATEGORIE]
id=##

zu allen CATEGORIE und CATEGORIE\subcategorie muessen
IDs vorhanden sein, andernfalls
step1, create import.sql fuer main CATEGORIE
if all CATEGORIE includes id
step2, create import.sql fuer subcategorie's


analyze filearea.ctl
use path i.e. Q:\DIV13\box_info
search groups,  if found path Q:\DIV13\BOX_INFO use this group
otherwise reduce path next lower level  Q:\DIV13
search groups, if found path Q:\DIV13 use this group
otherwise reduce path ... no more? stop, skip


program?   clipper ... mysql import is needed
php, without mysql import  (see search index process)  ???
search index uses perl ! (i:\apache2\apache\fbmaint\fbmaint.cmd
-> perl index.pl)
aber dafuer sitesearch 
cd \Apache\Apache2\sitesearch\admin
i:\apache\php\php -f mspider.php i:/apache/apache2/sitesearch/admin/spider2.txt
spider2.txt includes links to htm to analyse (update)

variant I

1. import areas       import areas to categories
     from mx.fareas.dat, flsearch.ctl, config, ....

2. admin module       sort/move from root areas to sub areas

variant II

1. import areas using "routing config" and 
     from mx.fareas.dat, flsearch.ctl, config, ....

    configure root areas, use directorys to parentid's
    see groups.lst

2. optional admin module

3. import areas2cat  (sync/save ipaths to categories)

4. import files using db info
    sync by compare filesize files.bbs

db: prefix_downloads_ipaths
  farea area, displayareaname, path, level, description, parentid, fbsize

zu 4)
insert into nuke_downloads_downloads values (
NULL,
'categorie#',        for downloads_ipath 1 to end, farea area -> #
'0',
'Filename',          file(path+filename)
 'url x2)',          basepath + path + filename
'description x3)',                  from files.bbs
'filedate yyyy-mm-dd hh:mm:ss x4)',  filedate(path+filename)
'Allfix',
'',
'0',
'uli',                             submitter
'0.0000',
'0',
'0',
'filesize x5)',                    filesize(path+filename)
'version x6)',                     from description?
''
);
ACHTUNG: PHP.INI config parameter 
 register_argc_argv = OFF -> ON !!!
 output_buffering=4096  -> 0


php program:
1. read own config
2. read phpnuke config
3. read phpnuke db config
4. rest

http or console mode  see mspider.php


CATEGOR.LST, BBS2WEB Output Directory List:
-------------------------------------------

 Verzeichnis von Q:\BBS2WEB\WWW

06ER     HTM         5.345 11.06.05   13:44
24000ZIP HTM        31.283 11.06.05   13:44
24000_V7 HTM         1.999 11.06.05   13:44
ALLFIX   HTM         4.244 11.06.05   13:44
ALLFIXUT HTM         4.275 11.06.05   13:44
[...]



CATEGORY.TXT, MySqlDump _prefix_Downloads_* Output:
---------------------------------------------------

-- MySQL dump 9.11
--
-- Host: localhost    Database: nuke
-- ------------------------------------------------------
-- Server version	4.0.20a-nt
--
-- Table structure for table `nuke_downloads_categories`
--

CREATE TABLE nuke_downloads_categories (
  cid int(11) NOT NULL auto_increment,
  title varchar(50) NOT NULL default '',
  cdescription text NOT NULL,
  parentid int(11) NOT NULL default '0',
  PRIMARY KEY  (cid),
  KEY cid (cid),
  KEY title (title)
) TYPE=MyISAM;
--
-- Dumping data for table `nuke_downloads_categories`
--
INSERT INTO nuke_downloads_categories VALUES (1,'Fidonet','Fidonet related Downloads',0);
INSERT INTO nuke_downloads_categories VALUES (3,'Webportale','Alles rund um Webinterface und
  Webportale\r\n',0);
INSERT INTO nuke_downloads_categories VALUES (4,'PHP-Nuke','PHP-Nuke Installations- und
  Upgrade Pakete. Auch Bundles.\r\n',3);
INSERT INTO nuke_downloads_categories VALUES (5,'Region24','Region 24 related files\r\n',1);
INSERT INTO nuke_downloads_categories VALUES (6,'Pointlisten','Region 24 Pointlisten\r\n',5);
INSERT INTO nuke_downloads_categories VALUES (7,'Scripts','Interessante Script Pakete\r\n',3);
INSERT INTO nuke_downloads_categories VALUES (45,'SERVICE','Globales Service Verzeichnis
  (Nodelisten, Diffs,u.a.)',1);
INSERT INTO nuke_downloads_categories VALUES (44,'FNEWSFRQ','Latest FIDONEWS Fido Newsletters'
  ,1);
INSERT INTO nuke_downloads_categories VALUES (43,'FNEWS','FIDONEWS Fido Weekly Newsletters',1);
INSERT INTO nuke_downloads_categories VALUES (42,'WI2','Webinterface II Project Files',3);
INSERT INTO nuke_downloads_categories VALUES (41,'Filelist','Filelisten',39);
INSERT INTO nuke_downloads_categories VALUES (40,'Box_Info','AMBROSIA60 Box Infos',39);
INSERT INTO nuke_downloads_categories VALUES (39,'FileBase','AMBROSIA60 FileBase Mirror',0);
INSERT INTO nuke_downloads_categories VALUES (46,'NODELIST','Latest Fidonet Nodelist',1);
INSERT INTO nuke_downloads_categories VALUES (47,'NODEDIFF','Latest Fidonet Nodediff',1);
INSERT INTO nuke_downloads_categories VALUES (48,'R24PNT','R24PNT and R24PNT_D, Alle Region24
  4D Pointlists Boss-Format',1);
INSERT INTO nuke_downloads_categories VALUES (49,'R24PFQ','Latest R24PNT and R24PNT_D',1);
INSERT INTO nuke_downloads_categories VALUES (50,'Z2PNT','Z2PNT, Alle Zone2 4D Pointlists
  Poss-Format',1);
INSERT INTO nuke_downloads_categories VALUES (51,'Z2PntFRQ','Latest Z2PNT Zone2 4D Pointlists
  Poss-Format',1);
INSERT INTO nuke_downloads_categories VALUES (52,'XPNTZ2DI','Z2PNT_D, Alle Zone2 4D PointDiffs
  Poss-Format',1);
INSERT INTO nuke_downloads_categories VALUES (53,'XPZ2DFRQ','Latest Z2PNT_D, Zone2 4D
  PointDiff Poss-Format',1);



FILEAREA.CTL, Maximus Filebase Definition File:
-----------------------------------------------
FileArea  BOX_INFO            
DownLoad Q:\DIV13\box_info
acs   Normal
Desc  AMBROSIA(60)_Box_Info
Type DateList
Type Free
End FileArea

[...]


'ROUTING' Config File for Areas to Categories relations


[EXCLUDE]                        Paths to exclude
Q:\X

[ROOT]                           no Areas to Root Path !

[FIDONET]                        Root Categorie, ParentID=0 (!)
parent=0
Q:\DIV13\netz244                 single, ungrouped areas
Q:\news
Q:\service
Q:\DIV13\region24
Q:\DIV27\I-BINKD
Q:\DIV13\P24CVRT
Q:\DIV13\fd
Q:\DIV13\mailutil
Q:\DIV13\fidosoft.244
Q:\DIV3\rules24
Q:\DIV3\RECLISTE
Q:\DIV27\I-UTIL
Q:\point
Q:\DIV13\FASTECHO
Q:\DIV27\ITRACK
Q:\div13\allfix
Q:\div13\allfixut
Q:\div27\ttick
Q:\SDS\ftsc   

[WEBPORTALE]                    2nd Main Categorie, ParentID=0 !
parent=0
Q:\DIV13\wi2                    only one single Area

[FILEBASE]                      3rd Main Categorie, ParentID=0 !
parent=0
Q:\DIV13                        rest of areas in Q:\DIV13 directory
Q:\filelist                    
Q:\DIV27                        rest of areas in Q:\DIV27 directory
Q:\DIV3                         rest of areas in Q:\DIV3  directory

[XP]                            Subgroup in Maingroup "FILEBASE" (see above)
parent=FILEBASE                 ParentID <> 0, ParentID unknown, use Category Name
Q:\xp                           XP (Crosspoint) Group Areas, starting with Directory Q:\XP\...

[PDN]                           same as XP, Subgroup in Maingroup "FILEBASE" (see above)
parent=FILEBASE                 ParentID <> 0, ParentID unknown, use Category Name
Q:\PDN                          PDN Group Areas, starting with Directory Q:\PDN\...

[...]



1. Step, import to TEMP_DOWNLOADS_TABLE
records after 1st step import

downloads_temp
--------------
area      parentid    parentname  path
--------- ----------- ----------- --------------------
EXCLUDE                           Q:\X\
ROOT
FIDONET    0
                      FIDONET     Q:\DIV13\netz244\
                      FIDONET     Q:\news\
                      FIDONET     Q:\service\
                      FIDONET     Q:\DIV13\region24\
                      FIDONET     Q:\DIV27\I-BINKD\
                      FIDONET     Q:\DIV13\P24CVRT\
                      FIDONET     Q:\DIV13\fd
                      FIDONET     Q:\DIV13\mailutil
                      FIDONET     Q:\DIV13\fidosoft.244
                      FIDONET     Q:\DIV3\rules24
                      FIDONET     Q:\DIV3\RECLISTE
                      FIDONET     Q:\DIV3\echolist
                      FIDONET     Q:\point
                      FIDONET     Q:\DIV13\FASTECHO
                      FIDONET     Q:\DIV27\ITRACK
                      FIDONET     Q:\div13\allfix
                      FIDONET     Q:\div13\allfixut
                      FIDONET     Q:\div27\ttick
                      FIDONET     Q:\SDS\ftsc
WEBPORTALE 0
                      WEBPORTALE  Q:\DIV13\wi2
FILEBASE   0
                      FILEBASE    Q:\DIV13
                      FILEBASE    Q:\filelist
                      ...



2nd step:  merge FILEAREA.CTL or FAREAS.DAT (Maximus) thru TEMP_DOWNLOADS_TABLE filter

area      parentid    parentname  path                  MX.Areaname     PHPNUKE-Areaname AccessLevel
--------- ----------- ----------- --------------------- --------------- ---------------- -----------
EXCLUDE                           Q:\X\                 SECURE                           PROTECTED
ROOT
FIDONET    0
                      FIDONET     Q:\DIV13\netz244\     NET244          NET244           NORMAL
                      FIDONET     Q:\news\              NEWS            NEWS             NORMAL
                      FIDONET     Q:\service\           SERVIC          SERVIC  x2)      NORMAL
                      FIDONET     Q:\DIV13\region24\    REGION24        REGION24         NORMAL
                      FIDONET     Q:\DIV27\I-BINKD\     I-BINKD         I-BINKD          NORMAL
                      FIDONET     Q:\DIV13\P24CVRT\     P24CVRT         P24CVRT x2)      NORMAL
                      FIDONET     Q:\DIV13\fd           FD              FD               NORMAL
                      FIDONET     Q:\DIV13\mailutil     MAILUTIL        MAILUTIL         NORMAL
                      FIDONET     Q:\DIV13\fidosoft.244 SOFT244         SOFT244          NORMAL
                      FIDONET     Q:\DIV3\rules24       RULES24         RULES24          NORMAL
                      FIDONET     Q:\DIV3\RECLISTE      RECLISTE        RECLISTE         NORMAL
                      FIDONET     Q:\DIV3\echolist      ECHOLIST        ECHOLIST         NORMAL
                      FIDONET     Q:\point              POINT           POINT            NORMAL
                      FIDONET     Q:\DIV13\FASTECHO     FASTECHO        FASTECHO         NORMAL
                      FIDONET     Q:\DIV27\ITRACK       ITRACK          ITRACK           NORMAL
                      FIDONET     Q:\div13\allfix       ALLFIX          ALLFIX           NORMAL
                      FIDONET     Q:\div13\allfixut     ALLFIXUT        ALLFIXUT         NORMAL
                      FIDONET     Q:\div27\ttick        TTICK           TTICK            NORMAL
                      FIDONET     Q:\SDS\ftsc           FTSC            FTSC    x3)      NORMAL
WEBPORTALE 0
                      WEBPORTALE  Q:\DIV13\wi2          WI2             WI2              NORMAL
FILEBASE   0
                      FILEBASE    Q:\DIV13
                      FILEBASE    Q:\filelist
                      ...
x1)                   FILEBASE    Q:\DIV13\MSDOS\       MSDOS           MSDOS            NORMAL
x1)                   FILEBASE    Q:\DIV13\PRINTER\     PRINTER         PRINTER          NORMAL
 

x1) new areas inserted by 'routing'  FILEBASE Q:\DIV13\ line to Maingroup FILEBASE
x2) names can be corrected in step 3 by PHPNUKE Admin Console
    i.e. correct SERVIC  to  SERVICE, correct P24CVRT to PNTLCVRT  etc.
x3) later move to Maingroup SDS, in PHPNUKE Admin Console



Maximus Access Levels:
-------------------------
Class Name  Privilege Level

Hidden           65535
SysOp            100
AsstSysOp        90
Clerk            80
Extra            70
Favored          60
Privil           50
Worthy           40
Normal           30
Limited          20
Demoted          10
Transient        0







© 2003-2024 by Ulrich Schroeter   00896