f13end / rssingest Goto Github PK

Configured RSSIngest and I'm having problems loading 
'http://www.google.com/trends/hottrends/atom/hourly'.  It gets to the point 
where it says "Starting to work with feed URL".

Any help would be appreciated.

Thank you.

Configured RSSIngest and I'm having problems loading 
'http://www.google.com/trends/hottrends/atom/hourly'.  It gets to the point 
where it says "Starting to work with feed URL".

Any help would be appreciated.

Thank you.

- What steps will reproduce the problem?
1. I have changed "$item_exists_sql" to check item title instead of id
2. The feed i'm fetching is a Google Alert feed


- What is the expected output? What do you see instead?

The output looks perfect, but nothing new is stored in the rssingest table

- What version of the product are you using? On what operating system?

Latest version, Ubuntu Server 12.04

- Please provide any additional information below.

    A Google Alert feed doesn't always put the newest entry on top, for some reason this seems to be a problem. When the first entry in the rss feed is determined to be an "existing item", any new entries below it do not get inserted to the rssingest table. 

    Everything looks perfect. When the script finds an existing title, it echo's "Not inserting existing item..", when the script finds the title does not exist it echo's "Inserting new item..". But nothing gets inserted. 

    There is nothing wrong with the code that connects to my sever. I know this because I have tried multiple other feeds with the same exact code -the same php file- and they work perfect. All new entries are inserted into the rssingest table.

I'm stumped lol.

Side Note - I am very grateful for your code, it is very well done :)

I am not a programmer, however i have followed your instruction. Am getting the 
following displayed on the IE page & the data is not getting imported. I am 
using the constructed URL as recommended by you. THere is No error on the 
server logs.

*/ //require_once("dbCon/dbcon.php"); $db_hostname="localhost"; 
$db_username="spiderco_stanley"; $db_password="network21"; 
$private_access_key="1621"; // Check a few bits and pieces 
if(isset($_GET['feed_url'])) { $feed_url = $_GET['feed_url']; } else { 
die("Need to pass the (consistent) 'feed url'"); } 
if(isset($_GET['access_key'])) { if($_GET['access_key']==$private_access_key) { 
echo "Access key correct, proceeding...

"; } else { die("wrong access key"); } } else { die("Need to pass the 
'access_key' URL parameter"); } try { /* query the database */ // $db = 
getCon(); $db = mysql_connect($db_hostname,$db_username,$db_password); if 
(!$db) { die("Could not connect: " . mysql_error()); } 
mysql_select_db("spiderco_rss", $db); echo "Starting to work with feed URL '" . 
$feed_url . "'"; /* Parse XML from http://feeds.bbci.co.uk/news/rss.xml */ 
//$RSS_DOC = simpleXML_load_file('http://feeds.bbci.co.uk/news/rss.xml'); 
libxml_use_internal_errors(true); $RSS_DOC = simpleXML_load_file($feed_url); if 
(!$RSS_DOC) { echo "Failed loading XML\n"; foreach(libxml_get_errors() as 
$error) { echo "\t", $error->message; } } /* Get title, link, managing editor, 
and copyright from the document */ $rss_title = $RSS_DOC->channel->title; 
$rss_link = $RSS_DOC->channel->link; $rss_editor = 
$RSS_DOC->channel->managingEditor; $rss_copyright = 
$RSS_DOC->channel->copyright; $rss_date = $RSS_DOC->channel->pubDate; //Loop 
through each item in the RSS document foreach($RSS_DOC->channel->item as 
$RSSitem) { $item_id = md5($RSSitem->title); $fetch_date = date("Y-m-j G:i:s"); 
//NOTE: we don't use a DB SQL function so its database independant $item_title 
= $RSSitem->title; $item_date = date("Y-m-j G:i:s", 
strtotime($RSSitem->pubDate)); $item_url = $RSSitem->link; echo "Processing 
item '" , $item_id , "' on " , $fetch_date , "
"; echo $item_title, " - "; echo $item_date, "
"; echo $item_url, "
"; // Does record already exist? Only insert if new item... $item_exists_sql = 
"SELECT item_id FROM rssingest where item_id = '" . $item_id . "'"; 
$item_exists = mysql_query($item_exists_sql, $db); 
if(mysql_num_rows($item_exists)<1) { echo "Inserting new item..
"; $item_insert_sql = "INSERT INTO rssingest(item_id, feed_url, item_title, 
item_date, item_url, fetch_date) VALUES ('" . $item_id . "', '" . $feed_url . 
"', '" . $item_title . "', '" . $item_date . "', '" . $item_url . "', '" . 
$fetch_date . "')"; $insert_item = mysql_query($item_insert_sql, $db); } else { 
echo "Not inserting existing item..
"; } echo "
"; } // End of form // } catch (Exception $e) { echo 'Caught exception: ', 
$e->getMessage(), "\n"; } ?>

hi, thanks a lot for this ingest.. :)
it helps me to build a new website that i build right now..

but i have some minor problem here.. 
it seems this ingest only work rss version 2

when i try different version of rss, it show up in my browser but it dont store 
to my database..

i have try to build up the code of that issue but none i get.. :(

thanks for your consider anyway... :)

____This script looks like it could be really useful for editing an RSS feed 
generated by Eventlist for Joomla:

http://methodist-central-hall.org.uk/index.php?option=com_eventlist&view=eventli
st&format=feed&type=atom

____I've set the code up like this (obviously the username/password/key are not 
the actual ones):

<?php

//require_once("dbCon/dbcon.php");

$db_hostname="localhost";
$db_username="thisismypasswork";
$db_password="thisismypassword";

$private_access_key="thisismyprivatekey";

// Check a few bits and pieces

if(isset($_GET['feed_url']))
{
    $feed_url = $_GET['feed_url'];
}
else
{
    die("Need to pass the (consistent) 'feed url'");
}


if(isset($_GET['access_key']))
{

    if($_GET['access_key']==$private_access_key)
    {
        echo "Access key correct, proceeding...<br/><br/>";
    }
    else
    {
        die("wrong access key");
    }
}
else
{
    die("Need to pass the 'access_key' URL parameter");
}


try
{
    /*  query the database */
    // $db = getCon();

    $db = mysql_connect($db_hostname,$db_username,$db_password);
    if (!$db)
    {
        die("Could not connect: " . mysql_error());
    }
    mysql_select_db("your_db", $db);

    echo "Starting to work with feed URL '" . $feed_url . "'";

    /* Parse XML from  http://sanctuary-westminster.org/server/in.php?feed_url=http://methodist-central-hall.org.uk/index.php?option=com_eventlist&view=eventlist&format=feed&type=atom */
    //$RSS_DOC = simpleXML_load_file('http://sanctuary-westminster.org/server/in.php?feed_url=http://methodist-central-hall.org.uk/index.php?option=com_eventlist&view=eventlist&format=feed&type=atom');

    libxml_use_internal_errors(true);
    $RSS_DOC = simpleXML_load_file($feed_url);
    if (!$RSS_DOC) {
        echo "Failed loading XML\n";
        foreach(libxml_get_errors() as $error) {
            echo "\t", $error->message;
        }
    }


    /* Get title, link, managing editor, and copyright from the document  */
    $rss_title = $RSS_DOC->channel->title;
    $rss_link = $RSS_DOC->channel->link;
    $rss_editor = $RSS_DOC->channel->managingEditor;
    $rss_copyright = $RSS_DOC->channel->copyright;
    $rss_date = $RSS_DOC->channel->pubDate;

    //Loop through each item in the RSS document

    foreach($RSS_DOC->channel->item as $RSSitem)
    {

        $item_id    = md5($RSSitem->title);
        $fetch_date = date("Y-m-j G:i:s"); //NOTE: we don't use a DB SQL function so its database independant
        $item_title = $RSSitem->title;
        $item_date  = date("Y-m-j G:i:s", strtotime($RSSitem->pubDate));
        $item_url   = $RSSitem->link;

        echo "Processing item '" , $item_id , "' on " , $fetch_date     , "<br/>";
        echo $item_title, " - ";
        echo $item_date, "<br/>";
        echo $item_url, "<br/>";

        // Does record already exist? Only insert if new item...

        $item_exists_sql = "SELECT item_id FROM rssingest where item_id = '" . $item_id . "'";
        $item_exists = mysql_query($item_exists_sql, $db);
        if(mysql_num_rows($item_exists)<1)
        {
            echo "<font color=green>Inserting new item..</font><br/>";
            $item_insert_sql = "INSERT INTO rssingest(item_id, feed_url, item_title, item_date, item_url, fetch_date) VALUES ('" . $item_id . "', '" . $feed_url . "', '" . $item_title . "', '" . $item_date . "', '" . $item_url . "', '" . $fetch_date . "')";
            $insert_item = mysql_query($item_insert_sql, $db);
        }
        else
        {
            echo "<font color=blue>Not inserting existing item..</font><br/>";
        }

        echo "<br/>";
    }

    // End of form //
} catch (Exception $e)
{
    echo 'Caught exception: ',  $e->getMessage(), "\n";
}
?>

____I then run the URL: 

http://server/rss.php?feed_url=http://methodist-central-hall.org.uk/index.php?op
tion=com_eventlist&view=eventlist&format=feed&type=atom&access_key=thisismypriva
tekey

_____And get the following message:

Access key correct, proceeding...

Starting to work with feed URL 
'http://methodist-central-hall.org.uk/index.php?option=com_eventlist''Failed 
loading XML Opening and ending tag mismatch: span line 497 and a Opening and 
ending tag mismatch: li line 494 and span Opening and ending tag mismatch: ul 
line 493 and li Opening and ending tag mismatch: span line 504 and a Opening 
and ending tag mismatch: li line 501 and span Opening and ending tag mismatch: 
div line 492 and li Opening and ending tag mismatch: span line 511 and a 
Opening and ending tag mismatch: li line 508 and span Opening and ending tag 
mismatch: div line 488 and li Opening and ending tag mismatch: span line 520 
and a Opening and ending tag mismatch: li line 517 and span Opening and ending 
tag mismatch: div line 487 and li Opening and ending tag mismatch: div line 486 
and ul Opening and ending tag mismatch: body line 62 and div Opening and ending 
tag mismatch: html line 2 and div Extra content at the end of the document

____I'm guessing I need to sanitize the input stream, but I'm not sure the best 
way to do that, or how to edit the script accordingly. 
Thanks for the work.

What steps will reproduce the problem?

1. Configured table and PHP file
2. Loaded page with RSS URL & access code

What is the expected output? What do you see instead?

Expect to see an import process but it hangs at "Starting to work with feed URL"

Please provide any additional information below.

RSS URL is http://www.google.com/trends/hottrends/atom/hourly

In the database the field item_date has the property "ON UPDATE 
CURRENT_TIMESTAMP " but I can't see why it should have that. I removed it 
because I want to update the data without changing the date.