I had to answer a question recently on how to quickly populate a database, and with IndexU Deluxe 1.2 it’s very very simple to do.
I started off with a list of URL’s obtained from an outside source. Where you get the URL’s don’t matter, as long as you have them. This can even be done with DMOZ Extractor slices.
First I made the categories I wanted in the database, since this database is actually alphabetical it’s very easy to do. Twenty seven categories (a-z plus numbers) later I was complete, then I added one more that I will delete later.
The last category was “unsorted”, which is where I imported all of my links to. A quick import and I’m left with 16000+ links, not too bad to start off with.
So off I went to the admin panel and I clicked on Fetch Meta, I made sure Keyword, Description and Title were checked and to make sure there were no issues during import I put one in the Total url/cycle box. Then I chose Yes to update all links in the database and clicked on Fetch.
A few hours later the fetching was done, I now had URL’s, titles, descriptions and keywords. An instant database almost for free!
So I went and looked at my database and of course noticed some sites that fetched titles like 404 not found and other server type errors. I figured I would just search for listings that had “404″ in the title and delete them. Then I saw more than 10 pages of sites with 404 errors. How could I make this easier to do?
I fired up phpMyAdmin and loaded up the MySQL statement below
DELETE FROM `idx_link` WHERE `title` = ‘404 not found’
After running that statement I still had some listings with 404 errors, but a lot less than before. So I proceeded to use a number of other common errors to delete the sites that obviously didn’t work.
Then I went in and manually searched for 404, 301, 500 etc in the title and delete all of those sites. I was left with 14,400 listings which is still a good number of listings for a brand new site.
At this point I updated category paths, updated number of links, updated the tag clouds and then exported my database via the indexu admin panel. Why through the admin panel?
Because exporting via the admin panel gives a nice output that can be imported easily into Excel.
Now I loaded up the database spreadsheet, copy and pasted the listings and sorted them by name. Now all I have to do is add the categories and reimport the spreadsheet and it’s done.
The total time (not including fetching the meta tags) was less than one hour. And I’m left with a fully populated website that is ready to go live.
