16 March 2016

Catastrophic Backtracking

RegExp Playground

This repository contains several regexp examples.

Examples

1. Catastrophic Backtracking

This is Java example of backtracking catastrophic shown in Coding Horror post This is a known behavior when using nesting repetition operators, that has been analyzed in stackoverlow, regular expressions dedicated sites.
It can be seen the exponential explosion that take to parse using regexp for a simple pattern:
(x+x+)y+
Details of this issue can be found on the above links.

Execute

mvn clean package
mvn exec:java -Dexec.mainClass="cl.devweb.regexp.exploit.CatastrophicBacktracking"

Rule of Thumb - Solution

The solution is simple. When nesting repetition operators, make absolutely sure that there is only one way to match the same match. If repeating the inner loop 4 times and the outer loop 7 times results in the same overall match as repeating the inner loop 6 times and the outer loop 2 times, you can be sure that the regex engine will try all those combinations.

OUTPUT

 mvn exec:java -Dexec.mainClass="cl.devweb.regexp.exploit.CatastrophicBacktracking"
[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building regexp-exploit 0.0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- exec-maven-plugin:1.4.0:java (default-cli) @ regexp-exploit ---
Found value: xxxxxxxxxxxxxy  timer=113.2 μs
Invalid match of 1 chars took 20.97 μs
Invalid match of 2 chars took 14.28 μs
Invalid match of 3 chars took 21.82 μs
Invalid match of 4 chars took 149.2 μs
Invalid match of 5 chars took 26.92 μs
Invalid match of 6 chars took 43.73 μs
Invalid match of 7 chars took 80.80 μs
Invalid match of 8 chars took 135.6 μs
Invalid match of 9 chars took 207.1 μs
Invalid match of 10 chars took 240.6 μs
Invalid match of 11 chars took 442.9 μs
Invalid match of 12 chars took 848.0 μs
Invalid match of 13 chars took 1.623 ms
Invalid match of 14 chars took 2.504 ms
Invalid match of 15 chars took 4.339 ms
Invalid match of 16 chars took 8.163 ms
Invalid match of 17 chars took 16.60 ms
Invalid match of 18 chars took 33.27 ms
Invalid match of 19 chars took 69.98 ms
Invalid match of 20 chars took 138.2 ms
Invalid match of 21 chars took 268.9 ms
Invalid match of 22 chars took 537.9 ms
Invalid match of 23 chars took 977.8 ms
Invalid match of 24 chars took 2.132 s
Invalid match of 25 chars took 4.260 s
Invalid match of 26 chars took 8.433 s
Invalid match of 27 chars took 17.18 s
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 34.977 s
[INFO] Finished at: 2016-03-16T12:38:59-03:00
[INFO] Final Memory: 9M/245M
[INFO] ------------------------------------------------------------------------
We can see taking a Thread Dump, where is spending (wasting?) most of the time:
"main" #1 prio=5 os_prio=31 tid=0x00007f8f9a006000 nid=0x1303 runnable [0x000000010a718000]
   java.lang.Thread.State: RUNNABLE
    at java.util.regex.Pattern$GroupHead.match(Pattern.java:4658)
    at java.util.regex.Pattern$Loop.match(Pattern.java:4785)
    at java.util.regex.Pattern$GroupTail.match(Pattern.java:4717)
    at java.util.regex.Pattern$Curly.match0(Pattern.java:4272)
    at java.util.regex.Pattern$Curly.match(Pattern.java:4234)
    at java.util.regex.Pattern$Curly.match0(Pattern.java:4272)
    at java.util.regex.Pattern$Curly.match(Pattern.java:4234)
    at java.util.regex.Pattern$GroupHead.match(Pattern.java:4658)
    at java.util.regex.Pattern$Loop.match(Pattern.java:4785)
    at java.util.regex.Pattern$GroupTail.match(Pattern.java:4717)
    at java.util.regex.Pattern$Curly.match0(Pattern.java:4272)
    at java.util.regex.Pattern$Curly.match(Pattern.java:4234)
    at java.util.regex.Pattern$Curly.match0(Pattern.java:4272)
    at java.util.regex.Pattern$Curly.match(Pattern.java:4234)
    at java.util.regex.Pattern$GroupHead.match(Pattern.java:4658)
    at java.util.regex.Pattern$Loop.match(Pattern.java:4785)
    at java.util.regex.Pattern$GroupTail.match(Pattern.java:4717)
    at java.util.regex.Pattern$Curly.match0(Pattern.java:4272)
    at java.util.regex.Pattern$Curly.match(Pattern.java:4234)
    at java.util.regex.Pattern$Curly.match0(Pattern.java:4272)
    at java.util.regex.Pattern$Curly.match(Pattern.java:4234)
    at java.util.regex.Pattern$GroupHead.match(Pattern.java:4658)
    at java.util.regex.Pattern$Loop.match(Pattern.java:4785)
    at java.util.regex.Pattern$GroupTail.match(Pattern.java:4717)
    at java.util.regex.Pattern$Curly.match0(Pattern.java:4272)
    at java.util.regex.Pattern$Curly.match(Pattern.java:4234)
    at java.util.regex.Pattern$Curly.match0(Pattern.java:4272)
    at java.util.regex.Pattern$Curly.match(Pattern.java:4234)
    at java.util.regex.Pattern$GroupHead.match(Pattern.java:4658)
    at java.util.regex.Pattern$Loop.match(Pattern.java:4785)
    at java.util.regex.Pattern$GroupTail.match(Pattern.java:4717)
    at java.util.regex.Pattern$Curly.match0(Pattern.java:4272)
    at java.util.regex.Pattern$Curly.match(Pattern.java:4234)
    at java.util.regex.Pattern$Curly.match0(Pattern.java:4272)
    at java.util.regex.Pattern$Curly.match(Pattern.java:4234)
    at java.util.regex.Pattern$GroupHead.match(Pattern.java:4658)
    at java.util.regex.Pattern$Loop.match(Pattern.java:4785)
    at java.util.regex.Pattern$GroupTail.match(Pattern.java:4717)
    at java.util.regex.Pattern$Curly.match0(Pattern.java:4272)
    at java.util.regex.Pattern$Curly.match(Pattern.java:4234)
    at java.util.regex.Pattern$Curly.match0(Pattern.java:4272)
    at java.util.regex.Pattern$Curly.match(Pattern.java:4234)
    at java.util.regex.Pattern$GroupHead.match(Pattern.java:4658)
    at java.util.regex.Pattern$Loop.match(Pattern.java:4785)
    at java.util.regex.Pattern$GroupTail.match(Pattern.java:4717)
    at java.util.regex.Pattern$Curly.match0(Pattern.java:4272)
    at java.util.regex.Pattern$Curly.match(Pattern.java:4234)
    at java.util.regex.Pattern$Curly.match0(Pattern.java:4272)
    at java.util.regex.Pattern$Curly.match(Pattern.java:4234)
    at java.util.regex.Pattern$GroupHead.match(Pattern.java:4658)
    at java.util.regex.Pattern$Loop.matchInit(Pattern.java:4801)
    at java.util.regex.Pattern$Prolog.match(Pattern.java:4741)
    at java.util.regex.Pattern$Start.match(Pattern.java:3461)
    at java.util.regex.Matcher.search(Matcher.java:1248)
    at java.util.regex.Matcher.find(Matcher.java:637)
    at cl.devweb.regexp.exploit.CatastrophicBacktracking.main(CatastrophicBacktracking.java:81)

TODO

  • add more examples :)

License

MIT . .

15 March 2016

Maven Cheat Sheet

Some reminder for Maven:

mvn clean package -DskipTests -Dmaven.javadoc.skip=true -Dhttps.protocols=TLSv1.2

1. Use a different settings.xml
mvn --settings ~/.m2/YourOwnSettings.xml clean install
or
mvn -s ~/.m2/YourOwnSettings.xml clean install

2. Jboss tasks:
 mvn jboss-as:deploy
 mvn jboss-as:redeploy
 mvn jboss-as:undeploy

3. Eclipse Maven clean:
(retired)
mvn eclipse:eclipse
mvn eclipse:clean
mvn install

4. set java version:
<project>
 [...]
 <build>
 [...]
 <plugins>
 <plugin>
 <groupId>org.apache.maven.plugins</groupId>
 <artifactId>maven-compiler-plugin</artifactId>
 <version>3.5.1</version>
 <configuration>
 <source>1.4</source>
 <target>1.4</target>
 </configuration>
 </plugin>
 </plugins>
 [...]
 </build>
 [...]
</project>

5. Missing Maven dependencies in Eclipse Project

   workaround to have an eclipse maven project


6. basic maven structure

src/main/java

 - src 
    - main 
          - java 
          - resources 
          - webapp 
     - test 
          - java 
          - resources 

- target

7. packaging:
<packaging>jar</packaging>

8. executing from maven:

add the following plugin to pom.xml:
<plugin>
 <groupId>org.codehaus.mojo</groupId>
 <artifactId>exec-maven-plugin</artifactId>
 <version>1.2.1</version>
 <configuration>
 <mainClass>com.myorganisationname.common.App</mainClass>
 </configuration>
<arguments> -Dlog4j.configurationFile=/tmp/log4j2.xml </arguments> <classpathScope>test</classpathScope> 
</plugin>

and run it with:

mvn exec:java

9. skipping test:

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19.1</version>
<configuration>
<skipTests>true</skipTests>
</configuration>
</plugin>
      or
 
      mvn install -DskipTests
 

 
     or
     mvn install -Dmaven.test.skip=true
 
 
10. Multiple Module Projects
<packaging>pom</packaging>

<modules>
<module>guide-ide-eclipse-site</module>
<module>guide-ide-eclipse-core</module>
<module>guide-ide-eclipse-module1</module>
</modules>


Add the parent to the POMs of the new modules:

 <parent>
 <groupId>guide.ide.eclipse</groupId>
 <artifactId>guide-ide-eclipse</artifactId>
 <version>1.0-SNAPSHOT</version>
 </parent>


mvn install
mvn eclipse:eclipse

http://maven.apache.org/plugins/maven-eclipse-plugin/reactor.html


11. web archetype
mvn archetype:generate -DgroupId=cl.devweb -DartifactId=wls_log_retrieval -DarchetypeArtifactId=maven-archetype-webapp -DinteractiveMode=false
12. copy dependencies:
 mvn dependency:copy-dependencies -DoutputDirectory=OUTPUT_DIR

13. tomcat
mvn tomcat7:deploy
mvn tomcat7:undeploy
mvn tomcat7:redeploy

14. verbose class
mvn test -DjvmArgs="-verbose:class"

15. web.xml-less
(servlet 3.0+)

<plugin>
 <groupId>org.apache.maven.plugins</groupId>
 <artifactId>maven-war-plugin</artifactId>
 <version>2.6</version>
 <configuration>
  <failOnMissingWebXml>false</failOnMissingWebXml>
 </configuration>
</plugin> 
 
 
16. weblogic
 
              mvn com.oracle.weblogic:weblogic-maven-plugin:deploy

           mvn com.oracle.weblogic:weblogic-maven-plugin:undeploy


            <plugin>
                <groupId>com.oracle.weblogic</groupId>
                <artifactId>weblogic-maven-plugin</artifactId>
                <version>12.1.3-0-0</version>
                <configuration>
                    <adminurl>t3://127.0.0.1:7001</adminurl>
                    <user>weblogic</user>
                    <password>welcome1</password>
                    <upload>true</upload>
                    <action>deploy</action>
                    <remote>true</remote>
                    <verbose>true</verbose>
                    <source>target/${project.artifactId}.${project.packaging}</source>
                    <name>${project.artifactId}</name>
                    <targets>AdminServer</targets>
                </configuration>
            </plugin>
 
17. debug flag
mvn -X
or
mvn -e

18. update (when it does not find jar)
mvn -U package

19. dependencies
mvn dependency:tree
     -Dverbose

 
20. skip javadoc
-Dmaven.javadoc.skip=true
  
 
21. assigning different https protocol version :
(to avoid: Received fatal alert: protocol_version)

-Dhttps.protocols=TLSv1.2

22. identifying excluding dependencies

mvn dependency:tree -Dverbose -Dincludes=log4j:log4j

i.e.:


[German@KDU foo-ejemplo (feature/ggonzalez/refactoring)]$ mvn dependency:tree -Dverbose -Dincludes=org.springframework:spring-core
[INFO] Scanning for projects...
[INFO]
[INFO] ---------------< cl.bar:foo-ejemplo >---------------
[INFO] Building foo-ejemplo 1.0.0
[INFO] --------------------------------[ war ]---------------------------------
[INFO]
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ foo-ejemplo ---
[INFO] cl.bar:foo-ejemplo:war:1.0.0
[INFO] \- cl.bar.plataforma.common:foo-ejemplo-lib:jar:1.0.1:compile
[INFO]    +- org.springframework:spring-context:jar:4.2.1.RELEASE:compile
[INFO]    |  +- org.springframework:spring-aop:jar:4.2.1.RELEASE:compile
[INFO]    |  |  \- (org.springframework:spring-core:jar:4.2.1.RELEASE:compile - omitted for duplicate)
[INFO]    |  +- org.springframework:spring-beans:jar:4.2.1.RELEASE:compile
[INFO]    |  |  \- (org.springframework:spring-core:jar:4.2.1.RELEASE:compile - omitted for duplicate)
[INFO]    |  +- org.springframework:spring-core:jar:4.2.1.RELEASE:compile
[INFO]    |  \- org.springframework:spring-expression:jar:4.2.1.RELEASE:compile
[INFO]    |     \- (org.springframework:spring-core:jar:4.2.1.RELEASE:compile - omitted for duplicate)
[INFO]    \- org.springframework:spring-jdbc:jar:4.2.1.RELEASE:compile
[INFO]       +- (org.springframework:spring-core:jar:4.2.1.RELEASE:compile - omitted for duplicate)
[INFO]       \- org.springframework:spring-tx:jar:4.2.1.RELEASE:compile
[INFO]          \- (org.springframework:spring-core:jar:4.2.1.RELEASE:compile - omitted for duplicate)
 

23. Add local dependencies directly 

<dependency>
 <groupId>com.sample</groupId>
 <artifactId>sample</artifactId>
 <version>1.0</version>
 <scope>system</scope>
 <systemPath>${project.basedir}/src/main/resources/yourJar.jar</systemPath>
</dependency>



        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-war-plugin</artifactId>
            <version>3.2.0</version>
            <configuration>
                <failOnMissingWebXml>true</failOnMissingWebXml>
                <webResources>
                    <resource>
                       <directory>${project.basedir}/lib</directory>
                       <targetPath>WEB-INF/lib</targetPath>
                    </resource>
                </webResources>
            </configuration>
        </plugin>   
  


24. purge local
to clean up old dependencies (when someone changes a library but didn't upgrade the version, so we have a stale version)

mvn clean dependency:purge-local-repository 



 
25. maven options for -XX

export MAVEN_OPTS="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/"


<plugins>
  <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <configuration>
      <argLine>-Xms512m -Xmx512m -XX:PermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:MaxPermSize=512m</argLine>
    </configuration>
  </plugin>
</plugins>


<properties>
<argLine>-Xms512m -Xmx512m -XX:PermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:MaxPermSize=512m</argLine>


26. sonar 

mvn sonar:sonar -Dsonar.host.url=http://127.0.0.1:9000 -Dsonar.login=d676f79d8ba83cdcf69f38f8471f0284ee242e09

for remote server:
settings.xml:

        <settings>
        <pluginGroups>
        <pluginGroup>org.sonarsource.scanner.maven</pluginGroup>
        </pluginGroups>
        <profiles>
        <profile>
        <id>sonar</id>
        <activation>
        <activeByDefault>true</activeByDefault>
        </activation>
        <properties>
        <!-- Optional URL to server. Default value is http://localhost:9000 -->
        <sonar.host.url>
        http://myserver:9000
        </sonar.host.url>
        </properties>
        </profile>
        </profiles>
       </settings>


27. javadoc 8 issues

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:2.9.1:jar (attach-javadocs) on project foo-api-rest: MavenReportException: Error while creating archive:
[ERROR] Exit code: 1 - /Users/German/development/workspace-oepe/foo-api-rest/src/main/java/cl/bar/plataformacomercial/foo/api/aop/HttpUtil.java:20: warning: no description for @param
[ERROR]      * @param request
[ERROR]        ^



This can be solved: https://devwebcl.blogspot.com/2016/01/jdk8-self-closing-element-not-allowed.html

28. maven deploy nexus
before configuring: settings.xml, Plugins, distributionManagement,  snapshotRepository

mvn clean deploy

https://www.baeldung.com/maven-deploy-nexus

29. install a library in the local repo

mvn install:install-file -DgroupId=cl.devweb.rbac -DartifactId=test -Dversion=1.0.0 -Dfile=test-rbac.jar -Dpackaging=jar -DgeneratePom=true

30. list system properties from Maven:

mvn help:system

    <build>
        <plugins>
            <plugin>
                <artifactId>maven-antrun-plugin</artifactId>
                <version>1.6</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>run</goal>
                        </goals>
                        <configuration>
                            <target>
                                <property environment="env" />
                                <echoproperties />
                            </target>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>



31.
wsdl jar
to be tested jar wsdl client proxies :

           
                      <plugin>
                <groupId>org.jvnet.jax-ws-commons</groupId>
                <artifactId>jaxws-maven-plugin</artifactId>
                <version>2.3</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>wsimport</goal>
                        </goals>
                        <configuration>
                            <wsdlDirectory>${basedir}/src/main/resources/wsdl/</wsdlDirectory>
                            <wsdlLocation>wsdl/endpoint_sample_wsdl.wsdl</wsdlLocation>
                            <sourceDestDir>${basedir}/src/main/java</sourceDestDir>



32.
list of licenses

mvn license:add-third-party


33. .m2 corrupt
Many times when there is a problem downloading jar files from any maven repository, the quick solution is to delete local reference at .m2.

34.
integration test (failsafe)

mvn verify


35.
Operating System (OS) profiles:

   <profiles>
     <profile>
        <id>Windows</id>
        <activation>
          <os>
            <family>Windows</family>
          </os>
        </activation>
        <properties>
          <script.extension>.cmd</script.extension>
        </properties>
      </profile>
      <profile>
        <id>unix</id>
        <activation>
          <os>
            <family>unix</family>
          </os>
        </activation>
        <properties>
          <!-- actually it's extension-less (instead of .sh)-->
          <script.extension></script.extension>
        </properties>
      </profile>      
    </profiles>



36.
creates classpath

mvn dependency:build-classpath


37.
add new repositories to settings.xml
many maven repos have modular repos for release, snapshots, thirdparties...

 <repositories>
   <repository>
       <id>maven-releases</id>
       <url>http://localhost:8081/repository/maven-releases/</url>
       <layout>default</layout>
       <releases>
           <enabled>true</enabled>
       </releases>
       <snapshots>
           <enabled>false</enabled>
       </snapshots>
   </repository>
   <repository>
       <id>maven-snapshots</id>
       <url>http://localhost:8081/repository/maven-snapshots/</url>
       <layout>default</layout>
       <releases>
           <enabled>false</enabled>
       </releases>
       <snapshots>
           <enabled>true</enabled>
       </snapshots>
   </repository>
   <repository>
       <id>thirdparty</id>
       <url>http://nexus.cl:8081/repository/thirdparty/</url>
       <layout>default</layout>
       <releases>
           <enabled>true</enabled> <!-- fir jdbc 1.7 from wls -->
       </releases>
       <snapshots>
           <enabled>false</enabled>
       </snapshots>
   </repository>
 </repositories>



38. hello world antrun
(and writing to a file a timestamp).

mvn antrun:run@tmp-echo

<execution>
    <id>tmp-echo</id>
    <phase>generate-resources</phase>
    <goals>
        <goal>run</goal>
    </goals>
    <configuration>
        <target>
            <tstamp>
                 <format property="last.updated" pattern="yyyy-MM-dd hh:mm:ss"/>
            </tstamp>
            <echo>Hello world!</echo>
            <echo message="${last.updated}"/>
            <concat destfile="log.txt" append="yes" fixlastline="yes">${last.updated}</concat>
            <concat destfile="log.txt" append="yes" fixlastline="yes">
                <path path="hola.txt"/>
            </concat>
        </target>
    </configuration>
</execution>


39. Owasp Dependency-check-maven
open source vulnerable libraries.

mvn dependency-check:check

            <plugin>
                <groupId>org.owasp</groupId>
                <artifactId>dependency-check-maven</artifactId>
                <version>5.2.4</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>check</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>


40. Modernizer Maven Plugin
yet another static analyzer

<plugin>
  <groupId>org.gaul</groupId>
  <artifactId>modernizer-maven-plugin</artifactId>
  <version>2.0.0</version>
  <configuration>
    <javaVersion>1.8</javaVersion>
  </configuration>
</plugin>



41. springboot startup

mvn spring-boot:run
 
 
42. javadoc plugin

mvn javadoc:javadoc

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-javadoc-plugin</artifactId>
                <version>2.9.1</version>
                <executions>
                    <execution>
                        <id>attach-javadocs</id>
                        <goals>
                            <goal>jar</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>


43. Overriding managed version maven dependency

parent pom spring boot 2.x
version 1.x can override a version easily by replacing properties of a library:

<properties>
 <json.version>20180130</json.version>
</properties>


however now with 2.0 it must be avoided to use parent starter and add manually each dependency. From 1, 2 & 3


<dependencyManagement>
 <dependencies>
 <dependency>

...
...
...


44. logging stdout
Maven can log with:

-l log file
--log-file log.txt

or using tee would be better (we can redirect stdout to a file)

mvn-all2.bat | tee /a log3.txt



45. profile command line
to execute a profile (by using the maven repository) use -P argument:

mvn clean package -P perfil

    <profiles>
        <profile>
            <id>perfil</id>
            <repositories>
                <repository>
                    <snapshots>
                        <enabled>false</enabled>
                    </snapshots>
                    <id>company-releases</id>
                    <name>company-releases</name>
                    <url>https://host:port/repository/maven-releases</url>
                </repository>


46. internal repository in pom.xml

<project>
  ...

 <repositories>
   <repository>
         <id>my-internal-site</id>
         <url>http://myserver/repo</url>
    </repository>
  </repositories>
  ...
</project> 

 

My Blog List

Blog Archive

Disclaimer

The views expressed on this blog are my own and do not necessarily reflect the views of Oracle.