Javascript to get source code from a webpage?

Steveire · 29-11-2005 7:56pm #1

I'm trying to write a piece of JavaScript which will use the source code of a webpage in some way. I don't seem to be able to access the source code though. As an example, i made this:

<html>
<body>
<script type="text/javascript">

window.location = "view-source:http://www.google.com"

var newText = sourceCodeOf("http://www.google.com")

document.write(newText.length)
</script>
</body>
</html>

and I imagine that the source of http://www.google.com gets put as a string into the variable newText, using an imaginary function sourceCodeOf. Any idea how that function might work? Is it even possible to do what i want? I thought it would be standard, but doesn't seem to be. I just put the pop-up bit in to convince myself that I can get to the source code in some way...

Thanks.

BrokenArrows · 29-11-2005 8:41pm

<html>
<body>

<HEAD>

<script language=JavaScript>
function ViewSource() {
var newtext = "view-source:" + "http://www.google.ie"
window.location = newtext
}
</script>

</HEAD>

<a href=javascript:ViewSource()>View Source</a>

</html>
</body>

Steveire · 29-11-2005 11:59pm

But that doesn't get the source into a string i can manipulate. Any ideas there? Google comes up blank, but that's only because I don't know what to look for. "Javascript source code" tends to throw up a lot.

OfflerCrocGod · 30-11-2005 12:10am

Use PHP and just return the source code of the website from your php script to your browser (you know......ajax*)

* without the x, treat it as text.

Karoma · 30-11-2005 12:18am

Can you explain what it is that you're trying to achieve?

seamus · 30-11-2005 12:34am

I typed in a reply and deleted it.

If, as I assume, you're attempting to read in the contents of http://www.google.com into a Javascript variable, and failing, I probably know why. Google examines the User-Agent header of the client requesting the page. If that header is non-existent or Google doesn't like it, you get back a blank page. Try another site (e.g. www.boards.ie) and if you can achieve this, then my explanation above is what the problem is.

Afaik, you can't specify HTTP headers when calling a page in Javascript. You'd have a much easier time with a real scripting language.

OfflerCrocGod · 30-11-2005 12:41am

As seamus said real scripting language[PHP]<?php
$site = $_GET["name"];
$fc=file($site);

foreach($fc as $line)
{
echo htmlspecialchars($line);
}
?>[/PHP]"http://localhost/php/webs.php?name=http://www.google.com"
Will return google's code no bother. Make sure you have the http part in - I presume you can also use fopen();

Steveire · 30-11-2005 12:49am

I'd try it with www.boards.ie if I knew what code i needed to type. Truth is, the sum total of everything I know about Javascript is in the Code box above. I know nothing about PHP other than I use it a lot to browse boards and the biki.

I want to be able to be able take certain links out of webpages, and rearrange them and manipulate them and the likes. First i need to figure out how to access the source.

EDIT: Alright, Jeez, I'll go off and learn PHP.

Odd: Firefox wouldn't let me select the contents of that box...

fopen() the C function will work?

OfflerCrocGod · 30-11-2005 12:56am

Steveire 2 days ago I'd never written two lines of PHP.......it's easy. Here is the web page to go with that PHP; it's a mess but it shows what to do.......I hope. <EDIT>Mistake in CODE, sorry

[PHP]<html>
<head>
<title>Test.</title>
<script type="text/javascript">
function ahah(url, target, delay) {
document.getElementById(target).innerHTML = 'waiting...';
if (window.XMLHttpRequest) {
req = new XMLHttpRequest();
} else if (window.ActiveXObject) {
req = new ActiveXObject("Microsoft.XMLHTTP");
}
if (req != undefined) {
req.onreadystatechange = function() {ahahDone(url, target, delay);};
req.open("GET", "http://localhost/php/webs.php?name="+url, true);
req.send("");
}
}

function ahahDone(url, target, delay) {
if (req.readyState == 4) { // only if req is "loaded"
if (req.status == 200) { // only if "OK"
document.getElementById(target).innerHTML = req.responseText;
} else {
document.getElementById(target).innerHTML="ahah error:\n"+req.statusText;
}
}
}
</script>
</head>

<body>
<a onclick="ahah('http://www.google.com', 'code', 0);">Click Here</a>
<div id="code"></div>
</body>
</html>[/PHP]

OfflerCrocGod · 30-11-2005 1:10am

seamus wrote:

I typed in a reply and deleted it.

If, as I assume, you're attempting to read in the contents of http://www.google.com into a Javascript variable, and failing, I probably know why. Google examines the User-Agent header of the client requesting the page. If that header is non-existent or Google doesn't like it, you get back a blank page. Try another site (e.g. www.boards.ie) and if you can achieve this, then my explanation above is what the problem is.

Can Javascript actually do that? Isn't that XSS? In otherwords a big no-no? What javascript code is used to access remote webpages? The small amount of Javascript I've looked at has never shown code like that, I'd be interested in learning it just in case

Steveire · 30-11-2005 10:03pm

I was trying to figure out a way to get a threadid number from this page, and then generate a random number close-ish to that and put it on the end of this:
http://www.boards.ie/vbulletin/showthread.php?threadid=

You've got yourself a random thread which is kinda recent. I can do all of it except the bit which gets that number. Nice dream i almost had. And I know how un reliable and thrown together it is, and that the numbers don't neccessarily follow iteratively, and you might not have access to the forum, and...
But that doesn't matter.

OfflerCrocGod · 01-12-2005 12:44am

This code returns all those numbers. By the way my above web code was totally wrong it's now right

Sorry.[PHP]<?php
//$board = $_GET["name"]; //use this is you want to use the above - now correct - ajax code.
$board = "http://www.boards.ie/index.php?filter=virgin";
$fc=file($board);
$findme = '&goto=newpost"';

foreach($fc as $line)
{
if(stripos($line, $findme)){
echo substr($line, 61, 10);
echo "<br />";
}
}
?>[/PHP]That returns a list like this
[PHP]2054856942
2054856941
2054856940
2054856938
2054856937
2054856936
2054856934
2054856933
2054856929
2054856928
2054856926
etc....[/PHP]Using that list should be very easy.

Steveire · 03-12-2005 2:44pm

Thanks for all that OfflerCrocGod. The thing is though, that i don't have a server to install PHP and MySQL on. Assuming I could install them on my desktop at home, that wouldn't be usable by anyone, because they'd only work when I'm online, right?

OfflerCrocGod · 03-12-2005 5:28pm

Steveire wrote:

Thanks for all that OfflerCrocGod. The thing is though, that i don't have a server to install PHP and MySQL on. Assuming I could install them on my desktop at home, that wouldn't be usable by anyone, because they'd only work when I'm online, right?

Yep that's right....I really can't see how this can be done using Javascript on its own. It could opens up XSS problems and that's why I doubt Javascript supports what you are trying to do; of course I could be wrong. So maybe asking on javascript specific forums would be an idea?

Javascript to get source code from a webpage?

Comments