Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[MSHARED-938] #60

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

[MSHARED-938] #60

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b89546e
[MSHARED-938]
nywitness Jul 24, 2020
faf0103
[MSHARED-938]
nywitness Jul 28, 2020
d2752c4
[MSHARED-938]
nywitness Jul 28, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions src/main/java/org/apache/maven/shared/utils/cli/CommandLineUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -280,11 +280,11 @@ public Integer call()
inputFeeder.start();
}

outputPumper = new StreamPumper( p.getInputStream(), systemOut );
outputPumper = new StreamPumper( p.getInputStream(), systemOut , streamCharset );
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Space before comma is not required.

outputPumper.setName( "StreamPumper-systemOut" );
outputPumper.start();

errorPumper = new StreamPumper( p.getErrorStream(), systemErr );
errorPumper = new StreamPumper( p.getErrorStream(), systemErr , streamCharset );
errorPumper.setName( "StreamPumper-systemErr" );
errorPumper.start();

Expand Down
22 changes: 22 additions & 0 deletions src/test/java/org/apache/maven/shared/utils/cli/CommandLineUtilsTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.fail;

import java.nio.charset.Charset;
import java.util.Arrays;
import java.util.HashMap;
import java.util.Locale;
Expand Down Expand Up @@ -168,4 +169,25 @@ private void assertCmdLineArgs( final String[] expected, final String cmdLine )
assertEquals( expected.length, actual.length );
assertEquals( Arrays.asList( expected ), Arrays.asList( actual ) );
}

@Test
public void testChineseEncodingIssue()
Copy link
Member

@michael-o michael-o Jul 28, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test is completely pointless:

  1. The echo will use the encoding supplied by the entire system. There is no guarantee that GBK is the system encoding.
  2. You never verify the output of th command to be what you expect.

What you need is an application that produces GBK bytes , those are read by Java with GBK into a String then you need to compare this value.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as i am concerned, the parameter commandLine of CommandLineUtils.executeCommandLineAsCallable is used to create a Process object, which is a result of Runtime.getRuntime().execute(). This execute() method uses different encoding depending on different system.

Any idea of producing GBK bytes using CommandLineUtils.executeCommandLineAsCallable? Or i can modify the test to use system encoding rather than using GBK.

Copy link
Author

@nywitness nywitness Jul 29, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By the way, this test will pass without this PR if system encoding is the same as the result of Charset.defaultCharset(). So the influence of this fix may not be very obvious.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as i am concerned, the parameter commandLine of CommandLineUtils.executeCommandLineAsCallable is used to create a Process object, which is a result of Runtime.getRuntime().execute(). This execute() method uses different encoding depending on different system.

Why do you think so? It uses the same encoding as the surrounding Java process does. You cannot change this really on Windows, on Unix you can pass LC_ALL to the env.

Any idea of producing GBK bytes using CommandLineUtils.executeCommandLineAsCallable? Or i can modify the test to use system encoding rather than using GBK.

You have two options:

  1. Modify file.encoding and set back in the finally block. Implies you read the output stream. I don't exactly know whether tests can run in parallel in the same JVM, this could break other tests.
  2. Write a simple Java program, put it in src/test/java, call the .class file with Java from within the test. It should use System.out as a byte-oriented stream which will write bytes according to GBK. Read those with the consumer and check when normalized back to UTF-16.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Write a simple Java program, put it in src/test/java, call the .class file with Java from within the test. It should use System.out as a byte-oriented stream which will write bytes according to GBK. Read those with the consumer and check when normalized back to UTF-16.

I tried producing gbk bytes with System.out.println(new String("金色传说".getBytes(), "GBK"). When comparing value in the cousumer, test passes on a windows-gbk platform but fails on a mac-utf-8 platform. Maybe it's not right to produce gbk bytes like that.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's wrong. You need to use System.out as an byte stream, not a char stream: byte[] bytes = "...".getBytes(encoding) then System.out.write(bytes).

throws Exception
{
Commandline commandline = new Commandline( "echo 金色传说" );
StreamConsumer err = new StreamConsumer() {
@Override
public void consumeLine( String line ) {

}
};
StreamConsumer out = new StreamConsumer() {
@Override
public void consumeLine( String line ) {
System.out.println( line );
}
};
CommandLineCallable commandLineCallable = CommandLineUtils.executeCommandLineAsCallable( commandline, null, out, err, 10, null, Charset.forName("GBK") );
commandLineCallable.call();
}
}