How to minimise data transfer

Sam Vimes

As we all know mobile data can be very expensive so we need to keep data transfer to the bare minimum. Usually I do it by putting a php script on a server and proxying any requests through it. The app sends just the required info to the script, which makes the request to the server, chops out the unnecessary html and just returns what the user needs.

But it only takes a few seconds for them to block your website's IP and kill your app. You can keep moving the scripts as the servers get blocked but that's not a permanent solution. Does anyone know of a better way to do it, preferably using J2ME?

anton

Could you be more specific? What problem are you trying to solve? Who are them who block your website's IP?

Sam Vimes

Them is the O2 website. The script is for sending web texts. A request is made to my website with the phone number, password, network, recipient number and message and my website logs into the o2 site, sends the text and reports back the number of messages you have left

anton

Ok, so you'll need some sort of proxying indeed. The obvious things that spring to mind are 'anonymizers', SOCKS proxies and Tor, try to google that and see if it suits you.

Sam Vimes

anton wrote: »

Ok, so you'll need some sort of proxying indeed. The obvious things that spring to mind are 'anonymizers', SOCKS proxies and Tor, try to google that and see if it suits you.

No I don't think they are suitable tbh. I can set it up so my website connects to another proxy server but then O2 can just block that and I'm back to where I started. The proxy site could end up blocking my IP too because they're not meant to be used like that.

And besides that, the requests contain passwords and personal messages so I don't want to forward them on to any third party. The passwords are in https requests but the messages aren't.

My preferred method is for the phone to connect directly to the O2 website and make the necessary requests without downloading all the unnecessary data but I don't know if that's possible......

anton

Sam Vimes wrote: »

No I don't think they are suitable tbh. I can set it up so my website connects to another proxy server but then O2 can just block that and I'm back to where I started. The proxy site could end up blocking my IP too because they're not meant to be used like that.

Well you'd have to have a list of proxy sites, not just one and use them in round robin fashion.

Sam Vimes wrote: »

And besides that, the requests contain passwords and personal messages so I don't want to forward them on to any third party. The passwords are in https requests but the messages aren't.

Well, if that's a concern for you, it rules out proxies alright.

Sam Vimes wrote: »

My preferred method is for the phone to connect directly to the O2 website and make the necessary requests without downloading all the unnecessary data but I don't know if that's possible......

Have a look at http layer then, you'll need to:

1. Authenticate

Probably involves sending POST request and getting back an authentication cookie of some sort. Maybe you won't even need to read entire HTTP response, just get the headers.


2. Push your message

Again, probably POSTing message along with some authentication information retrieved from step 1.

Sam Vimes

anton wrote: »

Have a look at http layer then, you'll need to:

1. Authenticate

Probably involves sending POST request and getting back an authentication cookie of some sort.

2. Push your message

Again, probably POSTing message along with some authentication information retrieved from step 1.

I've got this much working

anton wrote: »

Maybe you won't even need to read entire HTTP response, just get the headers.

That's what I'm trying to do


The problem is that as soon as I make the post request the entire page data is sent to the phone along with the cookies whether I read the http response or not. I've got this code:

	// Open an HTTP Connection object
	httpConn = (HttpConnection)Connector.open(url);
	// Setup HTTP Request to POST
	httpConn.setRequestMethod(HttpConnection.POST);

	httpConn.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");

	os = httpConn.openOutputStream();
	if(!params.equals("")){
		os.write(params.getBytes());
	}

	StringBuffer sb = new StringBuffer();
	is = httpConn.openDataInputStream();
	int chr;
	while ((chr = is.read()) != -1)
		sb.append((char) chr);
		return sb.toString();

but even if I comment out everything from StringBuffer sb = new StringBuffer(); onwards I can still see in wireshark that the whole page has been received. Any ideas?

anton

Well, I suppose it would suck in at least part of HTTP response body. Probably depends on particular phone/jre implementation as well.

What you could do is to use SocketConnection/SecureConnection along with small setSocketOption/RCVBUF/SNDBUF and implement (part of) HTTP protocol yourself.

BTW, didn't realize you are the Cabbage author , congrats it's seems to be quite popular on boards.

Sam Vimes

anton wrote: »

Well, I suppose it would suck in at least part of HTTP response body. Probably depends on particular phone/jre implementation as well.

What you could do is to use SocketConnection/SecureConnection along with small setSocketOption/RCVBUF/SNDBUF and implement (part of) HTTP protocol yourself.

I'll look into that, it seems like an awful lot of work though

anton wrote: »

BTW, didn't realize you are the Cabbage author , congrats it's seems to be quite popular on boards.

Thank you

anton

Not really, you'll probably only have to implement header parsing which isn't too hard for simple cases. And you probably can get some code to reuse on the net, have a look at that one for example https://meapplicationdevelopers.dev.java.net/mobileajax.html

Sam Vimes wrote: »

I'll look into that, it seems like an awful lot of work though

Dankoozy

don't use http it has too much overhead use 1 UDP packet + text compression or 5/6 bit encoding and a simple 1 byte ACK packet.

PHP has support for UDP but it might need to be compiled in

Sam Vimes

Dankoozy wrote: »

don't use http it has too much overhead use 1 UDP packet + text compression or 5/6 bit encoding and a simple 1 byte ACK packet.

PHP has support for UDP but it might need to be compiled in

I think you've missed something. Posting to a php script on my own website is fine, I can keep the data low on that, the problem is that the O2 website has blocked mine from accessing it so I can't forward on the request. I'm looking for a way for the phone to communicate directly with the O2 website but to receive only the headers in the response and not the hundreds of KB of HTML

mk6705

Sam Vimes wrote: »

I think you've missed something. Posting to a php script on my own website is fine, I can keep the data low on that, the problem is that the O2 website has blocked mine from accessing it so I can't forward on the request. I'm looking for a way for the phone to communicate directly with the O2 website but to receive only the headers in the response and not the hundreds of KB of HTML

Is this written in java?

Sam Vimes

Adro947 wrote: »

Is this written in java?

Yes it is

The Corinthian

I saw this thread yesterday and I got thinking...

Anyhow, I am not familiar with Cabbage (although I can see a lot of effort was put into it), but I wanted to through out a few ideas that in themselves are probably not directly usable, but may seed a practical solution for you.

Your main two objections to proxies are security and updating them.

For the first, one approach is that the client need not use a proxy for every transaction. Sensitive requests, such as logins, can be done directly, while heavier responses can be done using proxies. Additionally, you need not actually read all of the HTTP response on many requests - only enough to know if it has succeeded or failed.

Whether you know it or not, you actually already have a potential army of proxies with the people who download the desktop application. Of course, implementing a pseudo-P2P network is not going to be easy (not least of all as you will need to incentivise people to comply), but it would give you further control over any proxy network.

Naturally, you need to have a certain number of desktop proxies on-line for this to work. Too few and you'll end up with bottlenecks and, worse still, the poor sods who acted as proxies would get blocked. You probably know yourself if you have a critical mass for this or not - if not you need to find a way to encourage people to act as proxies.

Thus your activity flow could work something like this:

• M logs in directly to O - only reads enough of response to assess success/fail
• M checks a central site for any P online. P registers onto the central site whenever online. This central site could also act as a proxy between P and M.
• M requests any heavy data update from P, with session info
• P relays update any heavy data from O, compresses the response and sends it onto M
• M writes a message and sends directly to O - only reads enough of response to assess success/fail

(M = Mobile Client, P = Desktop Proxy, O = Operator Site)

Then there is the question of compression. I assume, given a proxy to process responses, you do more than simply compress, but actually scrape only what is relevant? Also, if you really want to compress that data, WAP's solution to this (WBXML) is quite relevant and a lot easier to implement than you would think.

Finally, whatever you do, I would program it in such a manner as to allow for remote updates.

Of course, the mobile networks will probably find ways to block you in time, but in my experience of them, for something more than an IP to block, 'in time' could be a year or two. Or more.

Sam Vimes

Thanks for all that, it's a lot to think about

I'm not sure how well using the desktop applications as proxies would work. As well as the considerations you mentioned, the only two requests made to the website are the login request and the "send message" request, neither of which I'd want being sent through unauthorised computers.

As for only reading part of the http response, as anton said it looks like I'll have to implement part of the http standard myself to do that because the phone seems to receive the whole lot in one go and then just feed it into the program as it's requested.

But this is all a moot point for the moment because I realised that the reason the program had stopped working over the O2 network wasn't because they'd blocked my website, it had something to do with how I was sending the request. It worked fine on other networks and over wifi but O2 rejected it for some reason. I've fixed it now so I won't have to deal with this problem at least for a while yet

Thanks for all the replies

matrim

Sam Vimes wrote: »

Thanks for all that, it's a lot to think about

I'm not sure how well using the desktop applications as proxies would work. As well as the considerations you mentioned, the only two requests made to the website are the login request and the "send message" request, neither of which I'd want being sent through unauthorised computers.

As for only reading part of the http response, as anton said it looks like I'll have to implement part of the http standard myself to do that because the phone seems to receive the whole lot in one go and then just feed it into the program as it's requested.

But this is all a moot point for the moment because I realised that the reason the program had stopped working over the O2 network wasn't because they'd blocked my website, it had something to do with how I was sending the request. It worked fine on other networks and over wifi but O2 rejected it for some reason. I've fixed it now so I won't have to deal with this problem at least for a while yet

Thanks for all the replies

Do you make the logon request for every message? If you know when the sessions timeout on the website it should be possible to store the session cookie and keep it for use again to save at least one request in some messages

Sam Vimes

matrim wrote: »

Do you make the logon request for every message? If you know when the sessions timeout on the website it should be possible to store the session cookie and keep it for use again to save at least one request in some messages

The time out is 24 hours on vodafone but something like 5 minutes on meteor. I think doing that would be a lot of hassle for very little saving tbh. It's charged at 2c per KB with a 99c per day ceiling so even a single text with both the login page and the "message sent" page returned would make it hit the ceiling. Unless it gets the entire transfer down to <1KB it makes the program useless, even 5KB would make it cost 10c, more than most networks charge for a text.

anton

I'd say don't bother minimizing data transfer, if anything make it use MORE traffic That'll make more people go for data plans/addons and will make internet connected apps more commonplace, which is good. I for example more than happy with my 4.99/250Mb addon on Meteor.

Sam Vimes

anton wrote: »

I'd say don't bother minimizing data transfer, if anything make it use MORE traffic That'll make more people go for data plans/addons and will make internet connected apps more commonplace, which is good. I for example more than happy with my 4.99/250Mb addon on Meteor.

I did consider that as a lot of people have add ons (I have the 250MB add on myself) and I think that's how Eirtext works but there are a few issues:

1. People on pre pay can't get add ons except on 3 and they don't want their first text each day to cost 99c
2. If I changed it now loads of people would doubtless be caught out by it and come and get me
3. It would make the whole process slower
4. Currently cabbage can be used anywhere in the world to send texts for about 2c. those add ons don't work in Thailand