Language Benchmarks

Anima

Hey,

I'm currently doing up a report for college. I've to compare C, C++ and Java. I can find the easy differences alright like architecture, portability, history, advatanges/disadvantages etc but I'm having trouble finding reliable performance benchmark comparisons of each language.

A lot of them seem biased with people just stating opinions as fact. Or if I do find something, its usually well out of date.

Does anyone know where I can find relevant information?

Martyr

There are certainly plenty of articles and conflicting views about whether Java is faster than C++.

IMHO it depends alot on the programmer and/or algorithm.

I'd expect well written C/C++ to run significantly faster than Java since its closer to the metal, we could put to test

yep, just an opinion.

i found 1 comparison here amongst alot!

Anima

Cheers, yeah I've found a bit more after a bit of searching. Also theres a wiki article on Java Performance thats quite big.

It seems with JIT and other enhancements to the JVM, Java runs just as well as C in some cases and better in others, thanks to its dynamic recompilation. Fairly interesting stuff

Cantab.

I've used the Xerces XML parser on both Java and C++. I was amazed to see that the Java performance was just as good on very large XML files. Granted, Java used slightly more memory, but nonetheless.

Regarding "benchmarks": it all depends how the code is written. Code optimised for C++ is not necessarily optimised for Java.

Hobbes

Anima wrote: »

Does anyone know where I can find relevant information?

Why not just write up your own tests rather then copying previous tests?

The short answer all of them are the fastest. Depending on what it is you are doing.

How you code the apps as well is a big factor.

Anima

Well of course you'll be using a language and its features to its strengths. I mean you're hardly going to be doing GUIs in C if you had the option of doing them in Java.

What I had in mind was raw data tests such as math libraries, data I/O etc. I guess I could run my own tests for these but you know... effort.

Cantab.

Anima wrote: »

Well of course you'll be using a language and its features to its strengths. I mean you're hardly going to be doing GUIs in C if you had the option of doing them in Java.

What I had in mind was raw data tests such as math libraries, data I/O etc. I guess I could run my own tests for these but you know... effort.

Getting back to the Xerces parser issue, I'd say I was limited more by disk access speed than by any language efficiencies. If I had a not so fast processor but the same disk speeds, I might have seen a difference between the two.

There are so many variables. The Java/C++ debate is not black and white.

Martyr

Could you write an operating system kernel in Java or .NET? probably not

Can you exploit SIMD features of PowerPC/x86 cpus like Altivec/SSE2?
..yes, i think so but the big question is whether Java would generate more efficient code than C/C++ for this purpose and i'd doubt it very much.

Anima wrote:

What I had in mind was raw data tests such as math libraries, data I/O etc. I guess I could run my own tests for these but you know... effort

Cryptography would also be a good test.
Perhaps compare OpenSSL performance with Java classes.

The thing is though, Java classes are nothing but machine code for the processor of system VM is running on.

For example on windows, Java classes are stored in DLL files..much the same as crypto functions in OpenSSL DLL are accessed by C/C++ program running on windows.

Hobbes

Martyr wrote: »

Could you write an operating system kernel in Java or .NET? probably not

Wrong. You can and they exist.

I think so but the big question is whether Java would generate more efficient code than C/C++ for this purpose and i'd doubt it very much.

Again it depends very much on how and what you are doing. JIT will create java code that performs as well or better then C++ depending on what you are coding and how long the application runs. (JIT can rewrite the assembly code on the fly to optimize).

For example on windows, Java classes are stored in DLL files.

No they are not.

NeverSayDie

Personally, I have very little interest and place very little faith in "language benchmarks". They seem to be almost always generated for and used in religious wars between fanboys, or idle blog posts.

If you're worried about performance, there's better things to focus on - regardless of the languages themselves, some non-trivial code put together by an expert in language X is quite likely to outperform the equivalent built in language Y by an amateur/intermediate developer.

Then there's the big picture stuff - ease of use, tools support, third party libraries, etc. Regardless of language X being "faster" than language Y, if an engineer can work more productively in Y, then they can spend the saved development time on better design, performance optimisation, debugging and profiling, etc, which will most likely end up with a faster/better Y-based solution.

Martyr wrote: »

Could you write an operating system kernel in Java or .NET? probably not

Actually that has been done, with .NET anyway. MS released the source code a while back for a research OS they built in managed code (variant of C# I think), it's on CodePlex somewhere - it was called Singularity, far as I recall.

Martyr

hobbes wrote:

Again it depends very much on how and what you are doing. JIT will create java code that performs as well or better then C++ depending on what you are coding and how long the application runs. (JIT can rewrite the assembly code on the fly to optimize).

How would this make Java faster? Do cpu's discriminate machine code generated by C/C++ compiler?

hobbes wrote:

No they are not.

Well atleast have the courtesy to tell me where they are?

After all, your code has to run on the processor, doesn't it? so where does it come from?

Martyr

NeverSayDie wrote:

Actually that has been done, with .NET anyway. MS released the source code a while back for a research OS they built in managed code (variant of C# I think), it's on CodePlex somewhere - it was called Singularity, far as I recall.

Thanks, thats interesting.

After a quick read, it seems that most of the low-level code was written using C and x86 assembly.

But yes, kernel written in Sing#, subset of Spec# which is extension of c# :eek:

Hobbes

Martyr wrote: »

How would this make Java faster? Do cpu's discriminate machine code generated by C/C++ compiler?

A C/C++ compiler compiles once into assembly. That is then executed on the processor.

Java (and .NET for that matter) compile into Bytecode. When the JVM loads the bytecode it converts it to assembly and runs that. Over time the JIT JVM can see where sections of the application require optimization and rewrite the assembly code while the application is running to make it faster.

You can't do this with C/C++ (you can with C sharp IIRC).

Prior to JIT the java language was interpreted (bytecode instructions) and was slow. Which is where it got the bad rap from. That is many years ago.

Well atleast have the courtesy to tell me where they are?

Class files are just stored on your machine as normal files. They are not stored in DLLs. They are just btyecode representations of your source code. They are not actually compiled into something that the operating system can read.

The JVM is what is written to be OS specific. That is what reads the class files and converts it to assembly to run on the OS/processor.

Martyr

A C/C++ compiler compiles once into assembly. That is then executed on the processor.

Java (and .NET for that matter) compile into Bytecode. When the JVM loads the bytecode it converts it to assembly and runs that. Over time the JIT JVM can see where sections of the application require optimization and rewrite the assembly code while the application is running to make it faster.

You can't do this with C/C++ (you can with C sharp IIRC).

Prior to JIT the java language was interpreted (bytecode instructions) and was slow. Which is where it got the bad rap from. That is many years ago.

OK, fair enough.

hobbes wrote:

The JVM is what is written to be OS specific. That is what reads the class files and converts it to assembly to run on the OS/processor.

I was talking about JVM.dll, java classes are here by the way..should have been more specific.

Martyr

depending on your version of java runtime environment installed.

i have 1.4.2 for xp here, under the directory:

C:\Program Files\Java\j2re1.4.2_05\bin

You'll see here alot of DLL files which contain the Java classes used by Java .CLASS files executed in a JVM.

Martyr

1 other thing there is api in jvm.dll called JNI_CreateJavaVM which allows you to resolve java methods or execute other java byte code.

but the actual code from what i can tell is stored in dll files, i'll look at this later.
for example, working with jpegs, zip archives, sound, authentication..etc this isn't code generated at runtime.

Hobbes

Martyr wrote: »

You'll see here alot of DLL files which contain the Java classes used by Java .CLASS files executed in a JVM.

The DLL files are for use in the JVM.

I don't think you understand the difference between the JVM and what a Java Class is.

Java Classes are not stored in DLLs, nor are they a DLL. The actual java classes used by the JVM (java.lang.*) are stored in the lib folder in a JAR file. Normally called RT.jar. These are OS independent.

The JVM on the other hand is OS dependent so would have DLL's for windows.

Martyr

hobbes wrote:

The DLL files are for use in the JVM.

yes, I know...did you not read my response?

hobbes wrote:

I don't think you understand the difference between the JVM and what a Java Class is.

uhm, yes i do know the difference.
there was a misunderstanding by you on what i was referring to, and this was because i wasn't specific enough.

i was making point that Java classes (runtime environment) on windows were stored in DLL files, not actual Java bytecode.

hobbes wrote:

Java Classes are not stored in DLLs, nor are they a DLL. The actual java classes used by the JVM (java.lang.*) are stored in the lib folder in a JAR file. Normally called RT.jar

i'm talking about runtime environment, the classes that a JVM would invoke on behalf of bytecode to perform an operating system task...thats what i mean.

i think if you atleast take the time to look at the data contained in these runtime environment files, you'll see i'm right.

Hobbes

Martyr wrote: »

I'm talking about runtime environment, the classes that a JVM would invoke on behalf of bytecode to perform an operating system task...thats what i mean.

The runtime environment (JRE) is the java class files + JVM. As I said the JVM will convert the Java class files into assembly language which is then run on the system.

The core API for Java are stored in class files in RT.jar. The JVMs only job is converting that into Assembly.

For example if your java class is making a socket call then the JVM will write the assembly language for the OS to make that call for that system.

JNI is just a means for a java command can make a direct call into a C/C++ API.

If there is any mistake it is on what you are classifying as what a class file is. It is like saying that C++ code is hardware because the instruction set is on the chip.

i think if you atleast take the time to look at the data contained in these runtime environment files, you'll see i'm right.

SCJP1.4 + SCJP5 and I've worked with Java for years. I think I should know how it works by now.

Martyr

well i'm glad you say this, hobbes because i was hoping an experienced java programmer could produce some code to test the theory of whether java is faster than C/C++.

i'm not a programmer at all.. but i'll bet if we were to both implement a cpu intensive algorithm, you writing it in java, and me the complete noob writing it in C/C++, i'd beat your java code.

want to test the theory?

or are ya just all talk? :P

marco_polo

Martyr wrote: »

well i'm glad you say this, hobbes because i was hoping an experienced java programmer could produce some code to test the theory of whether java is faster than C/C++.

i'm not a programmer at all.. but i'll bet if we were to both implement a cpu intensive algorithm, you writing it in java, and me the complete noob writing it in C/C++, i'd beat your java code.

want to test the theory?

or are ya just all talk? :P

So you have already decided on your conclusions then?

Writing fair benchmarks for managed vs compilled languages is a difficult task (I am not sure it is 100% possible) some of the JVM specific potential pitfalls are discussed here

http://www.ibm.com/developerworks/java/library/j-benchmark1.html

Martyr

yes, marco_polo, i have.

java is slower than c/c++

if we want to put it to test, lets do that.

i'm not programmer, just noob/amatuer.
hobbes has been working with java for years, so he knows what he's talking about.

i am willing to bet that anything i write using c/c++ will result in faster code than anything hobbes writes.

Martyr

actually, not just any code, but something with cpu intensive processing..

marco_polo

Martyr wrote: »

actually, not just any code, but something with cpu intensive processing..

Including the JVM startup time or not?

Martyr

no, you can load your program and perhaps have a button which records the start time, performs its processing, records end time then displays results.

marco_polo

Ok I have selected these two pieces of code as the benchmark. I'll let you do the rest.

Be sure and post up the results.

public class Loop
{
  public static void main(String[] args)
  {
    //10 000 billion iterations
    long time = System.currentTimeMillis();
    int REPEAT1 = 1000 * 1000;
    int REPEAT2 = 1000 * 1000 * 10;
    for (int i = 0; i < REPEAT1; i++)
    {
      for (int j = 0; j < REPEAT2; j++)
      {
        //do nothing
      }
    }
    time = (System.currentTimeMillis() - time)/1000;
    System.out.println("Time taken: (in seconds) " + time);
  }
}

#include <sys/time.h>
main(int argc, char *argv[])
{
  int i, j, REPEAT1, REPEAT2;
  struct timeval before, after;
  void *tzp;
  /*10 000 billion iterations*/
  tzp = 0;
  before = (struct timeval*) malloc(sizeof(struct timeval));
  after = (struct timeval*) malloc(sizeof(struct timeval));
  gettimeofday(&before, &tzp);
  REPEAT1 = 1000 * 1000;
  REPEAT2 = 1000 * 1000 * 10;
  for (i = 0; i < REPEAT1; i++)
  {
    for (j = 0; j < REPEAT2; j++)
    {
      //do nothing
    }
  }
  gettimeofday(&after, &tzp);
  printf("Time taken (in seconds): %ld\n",
                  (after->tv_sec-before->tv_sec));  
}

Martyr

are you saying that the loop does nothing, or do you want me to pick something?

a loop that does nothing isn't very useful..

marco_polo

Martyr wrote: »

are you saying that the loop does nothing, or do you want me to pick something?

a loop that does nothing isn't very useful..

I have never seen a 'useful' benchmark test. A loop that does nothing a billion timed is not useful but very CPU intensive.

What do I get if I win BTW?

Martyr

but when the JIT sees this code, or indeed a decent C/C++ compiler, it won't generate any machine code...it will not run the loops.

marco_polo

Martyr wrote: »

but when the JIT sees this code, or indeed a decent C/C++ compiler, it won't generate any machine code...it will not run the loops.

Damn, I was hoping you wouldn't have recent C\C++ compiler.

Martyr

well marco most compilers would be smart enough to see this one, even i saw it before having to compile it