Here is the error from dmesg I’ve got when trying to mount external HDD:

[345826.197335] EXT3-fs error (device sdd1): ext3_check_descriptors: Block bitmap for group 1920 not in group (block 0)!
[345826.198130] EXT3-fs (sdd1): error: group descriptors corrupted

Trying to mount as ext2 didn’t change anything:

[346348.544459] EXT2-fs (sdd1): error: ext2_check_descriptors: Block bitmap for group 1920 not in group (block 0)!
[346348.544470] EXT2-fs (sdd1): group descriptors corrupted

fsck didn’t want to help and was stopping after this:

% fsck.ext3 /dev/sdd1           
e2fsck 1.41.14 (22-Dec-2010)
fsck.ext3: Group descriptors look bad... trying backup blocks...
fsck.ext3: A block group is missing an inode table while checking ext3 journal for /dev/sdd1

What you can try to do to recover at least some data is to use backup superblock. Run mke2fs command (as root):

% mke2fs -n /dev/sdd1
mke2fs 1.41.14 (22-Dec-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
24420352 inodes, 97677200 blocks
4883860 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
2981 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
	4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968

Notice the numbers at the end, let’s try fsck with one of the superblock backups:

% fsck.ext3 -b 71663616 -y /dev/sdd1
e2fsck 1.41.14 (22-Dec-2010)
/dev/sdd1 was not cleanly unmounted, check forced.
Resize inode not valid.  Recreate? yes
 
Pass 1: Checking inodes, blocks, and sizes
...

It did work for me, good luck and let me know how did you do!

Here is the command:

find . -type f -printf '%TY-%Tm-%Td %TT %p\n' | sort -r | head

Introduction

I have a collection of 160 photos that I would like to montage into a movie. Each photo was done on a different day and the brightness of each photo varies. This will definitely not look good on the photo, so I’ve decided to “normalize” the brightness across the images. That is:

• Find a value for average brightness of my images.
• Brighten those too dark, and darken those that are too bright. Contrast should also be adjusted.

Calculate average image brightness

I have used utility called identify from ImageMagick package

% identify -format "%[mean] %f\n" *JPG | sort -rn 
44153.8 2011.06.22.JPG
37955.8 2011.06.11.JPG
#...many more...
23019.2 2011.06.01.JPG
21256.8 2011.06.06.JPG

-format option allows for specifying what kind of information about the image we want to output. In this case I went for average brightness – %[mean] and filename – %f, see the full documentation if you’re interested in other possibilities.

The difference between top and bottom photos is quite big: 21256.8 for 2011.06.06.JPG and 44153.8 for 2011.06.22.JPG – see the photos below.

Now let’s calculate the average value of brightness – awk to the rescue!

% identify  -format "%[mean] %f\n" *JPG | sort -rn | awk 'BEGIN {FS=" "} { sum += $1; } END { printf "%s",sum/NR}' 
29481.9

Equalize the brightness

So I need to bring all the photos to the mean brightness of about 29481.9. The best option I’ve found to do that is to use -sigmoidal-contrast from ImageMagick. I don’t know what values for -sigmoidal-contrast I need to use, to get from one brightness to another, so I wrote a simple script that will try to do that using binary search algorithm. It’s really very quick & dirty script but it does the job. The main loop is as follows:

do {
$current = ($min + $max) / 2;
$brightness = adjust($current, $file);
$diff = $target - $brightness;
if($diff > 0) {
//we need to make it lighter
$min = $current;
} else {
//we need to make it darker
$max = $current;
}
$i++;
} while(abs($diff) > 200);

adjust function basically calls external convert utility to perform the adjustment:

convert $file -sigmoidal-contrast '$value,0%' $tmpfile

and then checks and returns the brightness after conversion. I use -sigmoidal-contrast option to increase brightness and +sigmoidal-contrast to decrease. Now, that’s much better:

Here is the complete script – as I’ve said: quick & dirty job.

Recently I had to locate the widest image in the set of photos in a directory. Here is how you can easily do it with identify command, from ImageMagick suit:

% identify -format '%w %f\n' * | sort -rn | head -n1
563 2010.12.30.JPG

You can see in the output that the widest image is 563 pixels wide and the filename is 2010.12.30.JPG.

mogrify is the tool you are looking for – it comes with the great imagemagick package and it’s designed especially to be used for bulk operations.
To resize all images in current directory to 600 pixels wide (width):

% mogrify -resize 600 *

The same but 600 height:

% mogrify -resize x600 *

I have a collection of related photos stored on my PC. The filenames were “ugly” – standard file names coming from the photo camera. I would like to rename them into a file containing the date when the photo was shoot, e.g. 2011.07.21.jpg. There is an excellent commandline tool to do just that – renrot. It’s packaged for Debian & Ubuntu so to install it, I’ve simply run:

apt-get install renrot

Then, in the directory that contains my photos I’ve run:

renrot -n "%Y.%m.%d" *

-n option allows for specifying the template used for renaming the files – it is year, month and day separated with dots in my case. If the tool creates more files, you can remove the originals:

rm *orig

Refer to manual page for the full options!

Introduction

In this blog we’ll be looking at how to use hibernate tools with just ant. This may be useful if you need to plug some hibernate tasks into your build system or have them scripted in any way. We will create a stand-alone ant build file with the minimum number of required libraries.

Libraries

I’ve extracted hibernate tools jar from eclipse hibernate plugin. You can find it in your eclipse installation directory, under plugins:
plugins/org.hibernate.eclipse_X.X.X.X/lib/tools. Take hibernate-tools.jar and freemarker.jar. From plugins/org.hibernate.eclipse_X.X.X.X/lib/hibernate grab more jars:

• dom4j-1.6.1.jar
• log4j-1.2.15.jar
• slf4j-api-1.5.8.jar
• slf4j-log4j12-1.5.8.jar

You will also need org.apache.commons.logging.jar and JDBC driver. I use MySQL, so I’ve added mysql-connector-java-5.1.13.jar as well. Put all the jars into “lib” directory. Create other directories: classes, src and generated. Or don’t bother – download the whole lot as I’ve set it up.

Java entities

We will use two sample entity classes (POJOs) for that example: Password and Resource. A Resource can contain many Passwords.

public class Password implements Serializable {
        private static final long serialVersionUID = 1L;
        private int id;
        private String password;
 
        // many-to-one association to Resource
        private Resource resource;
 
        public Password() {
        }
        public int getId() {
                return this.id;
        }
        public void setId(int id) {
                this.id = id;
        }
        public String getPassword() {
                return this.password;
        }
        public void setPassword(String password) {
                this.password = password;
        }
        public Resource getResource() {
                return this.resource;
        }
        public void setResource(Resource resource) {
                this.resource = resource;
        }
}

public class Resource implements Serializable {
        private static final long serialVersionUID = 15146339L;
        private int id;
        private String name;
 
        public Resource() {
        }
        public int getId() {
                return this.id;
        }
        public void setId(int id) {
                this.id = id;
        }
        public String getName() {
                return this.name;
        }
        public void setName(String name) {
                this.name = name;
        }
}

Hibernate configuration

Next, let’s create simple hibernate.cfg.xml…

<?xml version='1.0' encoding='utf-8'?>
<!DOCTYPE hibernate-configuration PUBLIC
"-//Hibernate/Hibernate Configuration DTD//EN"
"http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd">
 
<hibernate-configuration>
<session-factory>
  <property name="hibernate.connection.driver_class">com.mysql.jdbc.Driver</property>
  <property name="hibernate.connection.url">jdbc:mysql://localhost/hibernatetutorial</property>
  <property name="hibernate.connection.username">root</property>
  <property name="hibernate.connection.password">****</property>
  <property name="hibernate.connection.pool_size">10</property>
  <property name="show_sql">true</property>
  <property name="dialect">org.hibernate.dialect.MySQLDialect</property>
  <property name="hibernate.hbm2ddl.auto">update</property>
  <!-- Mapping files -->
  <mapping resource="password.hbm.xml"/>
  <mapping resource="resource.hbm.xml"/>
</session-factory>
</hibernate-configuration>

…and mapping files for password and resource.

<?xml version="1.0"?>
<!DOCTYPE hibernate-mapping PUBLIC
"-//Hibernate/Hibernate Mapping DTD 3.0//EN"
"http://hibernate.sourceforge.net/hibernate-mapping-3.0.dtd">
 
<hibernate-mapping>
        <class name="net.opensesam.entity.Password" table="password">
                <id name="id" type="long" column="ID" />
                <many-to-one name="resource_id" class="net.opensesam.entity.Resource" />
                <property name="password">
                        <column name="password" />
                </property>
        </class>
</hibernate-mapping>

<?xml version="1.0"?>
<!DOCTYPE hibernate-mapping PUBLIC
"-//Hibernate/Hibernate Mapping DTD 3.0//EN"
"http://hibernate.sourceforge.net/hibernate-mapping-3.0.dtd">
 
<hibernate-mapping>
  <class name="net.opensesam.entity.Resource" table="resource">
   <id name="id" type="long" column="ID" />
   <property name="name">
     <column name="name" />
   </property>
 </class>
</hibernate-mapping>

Ant build file

Finally we can build build.xml! We’ll first add paths to the required jars, then define new task: hibernatetool. We can then use our new task via <hibernatetool> tag – see the code below, it’s pretty self-explanatory.

<project name="MyProject" basedir="." default="default">
<path id="toolslib">
<path location="lib/dom4j-1.6.1.jar" />
<path location="lib/freemarker.jar" />
<path location="lib/hibernate3.jar" />
<path location="lib/hibernate-tools.jar" />
<path location="lib/log4j-1.2.15.jar" />
<path location="lib/org.apache.commons.logging.jar" />
<path location="lib/slf4j-api-1.5.8.jar" />
<path location="lib/slf4j-log4j12-1.5.8.jar" />
<path location="lib/mysql-connector-java-5.1.13.jar" />
</path>
 
<taskdef name="hibernatetool" 
         classname="org.hibernate.tool.ant.HibernateToolTask" 
         classpathref="toolslib" />
 
<target name="default">
<hibernatetool destdir="./generated">
 <classpath>
  <path location="." />
  <path location="./classes" />
 </classpath>
 
 <configuration configurationfile="hibernate.cfg.xml"/>
 <hbm2ddl export="false" outputfilename="sql.ddl"/>
</hibernatetool>
</target>
</project>

We are running above hbm2ddl task to export database schema into ./generated/sql.ddl file. Other tasks you can use are:

• hbm2java – generate POJOs
• hbm2dao – generate DAOs
• hbmtemplate – generate any custom code you want using templates (aka scaffolding)

Refer to the documentation for more details.

Hibernate annotations

If you prefer to use hibernate annotations, you need to do few changes. Add two extra jars:

• hibernate-annotations-3.5.6-Final.jar
• hibernate-jpa-2.0-api-1.0.0.Final.jar

Change <configuration> tag into <annotationconfiguration>:

 <annotationconfiguration configurationfile="hibernate.cfg.xml"/>

Your hibernate.cfg.xml will obviously need to contain pointers to annotated classes, e.g.:

  <mapping class="net.opensesam.entity.Password" />

I have it configured with the open source project “Open Sesame” – you can see the whole configuration in git repository.

You can also download a full source code for the non-annotated example.

Introduction

Remember me functionality is a mechanism that lets users log in automatically, even when they re-start their browser. In other words: after their session cookie is deleted by the browser (which is normally after closing the browser). This is implemented by setting up another cookie that will live in the browser’s database for a longer period (e.g. weeks). Spring out of the box offers two implementations of this feature. This tutorial presents two different approaches to cookie creation: hash-based token and persistent token. We will only look into the remember-me part of the Spring Security configuration, if you need more information look at basic setup.

Hash-based token

Implemented by org.springframework.security.web.authentication.rememberme.TokenBasedRememberMeServices, it creates a base64 encoded cookie that contains:

• username – you will be logged in as this username, if cookie is confirmed to be authentic.
• expiryTime – Spring will treat cookie as invalid after expiry time. The cookie is set to be valid for 2 weeks when created.
• hash – md5 hash of 2 values above, user’s password, and a key.

Hash is used to confirm that cookie is valid – e.g. no-one has tried to tweak the cookie manually. A key is a very important variable here – we set it in Spring’s configuration and it should be a unique, secret for our application. As nobody knows what key we have picked, no one will be able to generate valid hash. The above also means that:

• There is no extra information stored on the server side.
• If cookie is intercepted by an attacker, it can be used to log in as valid user.
• Username is exposed in cookie and can be read
• Cookie becomes invalid when user changes password.

The configuration itself is trivial, simply add one line with <remember-me> tag to security.xml (again, refer to the previous post for basics):

<http auto-config="true" >
		<remember-me key="...verylonganduniquekey..." />
		<intercept-url pattern="/*" access="ROLE_USER" />
	</http>

This automatically adds a new checkbox to the standard login form.

Once you login, notice additional cookie created: SPRING_SECURITY_REMEMBER_ME_COOKIE.

To test functionality, delete JSESSIONID cookie and refresh. Normally that would effectively result in a logging out but with our “remember me” cookie, new JSESSIONID is created straight away.

Persistent token

A better solution to remember me cookie is to use totally random string that will be stored as a cookie and has no meaningful data in itself. The disadvantage is that it requires storing some information on the server side. When token is presented from user’s browser, it is matched against a record in table with all tokens. Our Spring configuration will change only a bit.

<http auto-config="true" >
		<remember-me data-source-ref="dataSource" />
		<intercept-url pattern="/*" access="ROLE_USER" />
	</http>

We will also need a table “persistent_logins” to store tokens. For MySQL you can use following SQL.

CREATE TABLE IF NOT EXISTS `persistent_logins` (
  `username` VARCHAR(64) DEFAULT NULL,
  `series` VARCHAR(64) NOT NULL DEFAULT '',
  `token` VARCHAR(64) DEFAULT NULL,
  `last_used` datetime DEFAULT NULL,
  PRIMARY KEY (`series`)
)

As you can see there are actually two new pieces there: series and token. Both are random strings and both are passed back to user and stored in the cookie. The idea here is that token will be changed on every “remember me” login, so when a user comes with valid series but invalid token – the chances are that the cookie has been stolen and used for authentication! We could not prevent stealing the cookie but we can at least:

• warn user
• invalidate all tokens for current user

And this is exactly what Spring Security is doing.

Sometimes it might be very convenient to be able to debug application deployed on web server. This tutorial presents how to do it, if the application is deployed using Jetty plugin for Maven or Tomcat servers.

Jetty Maven plugin

Set the following MAVEN_OPTS to the following:

export MAVEN_OPTS='-Xdebug -Xrunjdwp:transport=dt_socket,address=4000,server=y,suspend=y'

Run Jetty Maven plugin:

mvn jetty:run

Sample setting of the development environment can be found here: Eclipse environment for convenient development of Java web application using Maven, Spring, Struts2, and Hibernate running on Jetty server.

Tomcat

Use default options of Tomcat which are

-agentlib:jdwp=transport=dt_socket,address=8000,server=y,suspend=n

or set JPDA_OPTS using the following command:

export JPDA_OPTS='-agentlib:jdwp=transport=dt_socket,address=4000,server=y,suspend=y'

Run Tomcat:

catalina jpda start

Arguments

Here is list of parameters for the debugging.

• Xdebug – Start the JVM and listen for debugging connections.
• Xrunjdwp – The remote debugging connection parameters.
• transport=dt_socked – Use sockets.
• address=4000 – The port to connect to or the shared memory address to use.
• server=y – Start in server mode.
• suspend=y – If ‘n’ don’t wait for a debugger to be attached. If ‘y’ your application will lunch only when you run your debugger.

Debugging in Eclipse

Go to: Debug -> Debug Options…
Select: Remote Java Application
Select project
Use the same port as defined in ‘address’

Click: Debug

If ‘suspend=y’ then your application should start. At this stage when you put any break point in Eclipse, your application should stop there.

References

Please refer to:

• Debugging with the Maven Jetty Plugin in Eclipse
• How do I configure Tomcat to support remote debugging

Introduction

In the tutorial below we will look at a basic integration of a Java web application with Spring Security 3. In this particular scenario we’ll be integrating Spring Security with existing web application that uses Struts & Spring. However, similar steps can be applied to any other web framework you may be using. We will also look at one of the core concepts Spring Security – AccessDecisionManager.

Basic integration

We will start with an application that has no security implemented. I’ve used this setup as my base. It already uses maven, so the first thing we do is to add a new repository:

<repository>
			<id>org.springframework.security</id>
			<name>Spring Security</name>
			<url>http://maven.springframework.org/snapshot</url>
		</repository>

and bunch of new JARs:

<dependency>
			<groupId>org.springframework.security</groupId>
			<artifactId>spring-security-core</artifactId>
			<version>3.0.6.CI-SNAPSHOT</version>
		</dependency>
		<dependency>
			<groupId>org.springframework.security</groupId>
			<artifactId>spring-security-config</artifactId>
			<version>3.0.6.CI-SNAPSHOT</version>
		</dependency>
		<dependency>
			<groupId>org.springframework.security</groupId>
			<artifactId>spring-security-web</artifactId>
			<version>3.0.6.CI-SNAPSHOT</version>
		</dependency>

Spring Security is implemented as a filter – it can be plugged into the application without any changes. A request will be intercepted by DelegatingFilterProxy and authentication will take place, as defined in Spring security XML file. We will add filter to the top of web.xml and trigger it for all URLs (url pattern set to /*).

<filter>
		<filter-name>springSecurityFilterChain</filter-name>
		<filter-class>org.springframework.web.filter.DelegatingFilterProxy</filter-class>
	</filter>
 	<filter-mapping>
		<filter-name>springSecurityFilterChain</filter-name>
		<url-pattern>*</url-pattern>
	</filter-mapping>

For the very basic integration, Spring Security will require only little configuration that we’ll put into security.xml file.

<?xml version="1.0" encoding="UTF-8"?>
<beans:beans xmlns="http://www.springframework.org/schema/security"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:beans="http://www.springframework.org/schema/beans"
	xsi:schemaLocation="
		http://www.springframework.org/schema/beans 
		http://www.springframework.org/schema/beans/spring-beans.xsd
		http://www.springframework.org/schema/security 
		http://www.springframework.org/schema/security/spring-security-3.0.xsd
	">
	<http auto-config="true">
		<intercept-url pattern="/*" access="ROLE_USER" />
	</http>
	<authentication-manager alias="authenticationManager">
		<authentication-provider>
			<user-service>
				<user authorities="ROLE_USER" name="guest" password="test" />
			</user-service>
		</authentication-provider>
	</authentication-manager>
</beans:beans>

We are defining here that access to any resource (intercept-url pattern=”/*”) should be allowed only for users that have authority called ROLE_USER. Normally user credentials will need to be stored in a central place – typically a database or LDAP server. Above we are hard-coding an authentication-provider with one user (guest/test) that has authority ROLE_USER.

Finally, we need to tell Spring about security.xml. Since our application already uses Spring, we have defined in web.xml contextConfigLocation parameter already – it points to our beans definition (applicationContext.xml). We will add security.xml in the same place:

<context-param>
		<param-name>contextConfigLocation</param-name>
		<param-value>
		/WEB-INF/resources/applicationContext.xml
		/WEB-INF/resources/security.xml
		</param-value>
	</context-param>

When we build the application now, any URL should automatically generate following login page:

Access decision manager

Spring Security deals (or helps you deal with) two security concepts: first authentication and then authorization. Authentication filters are run first – they will make sure that a user is who he says he is. This is usually done by checking user credentials, e.g., username and password). Let’s skip authentication step for now and talk about authorization. Authorization takes care of deciding if authenticated user (meaning that authentication must precede authorization) should be allowed to access a resource.

Spring Security supports authorization with the help of AccessDecisionManager interface. AccessDecisionManagers in turn will use one or more AccessDecisionVoters. Each voter will make the decision about granting or denying the access for given user to given resource. Access decision manager will aggregate the votes and issue final decision. Voters can return: ACCESS_GRANTED,ACCESS_DENIED but also ACCESS_ABSTAIN if a voter can’t or doesn’t want to make a decision. Managers must make the final decision – user is either granted access or not. The “not granting the access” part is done by throwing an exception – usually AccessDeniedException. In the default configuration that we’ve created, Spring uses implementation of AccessDecisionManager called AffirmativeBased. It will grant the access if any voter returns ACCESS_GRANTED. The default voters are

• RoleVoter – checks GrantedAuthority (remember user tag with authorities=”ROLE_USER”) against ConfigAttribute (that comes from intercept-url pattern=”/*” access=”ROLE_USER”)
• AuthenticatedVoter – can check if user IS_AUTHENTICATED_FULLY, IS_AUTHENTICATED_REMEMBERED or IS_AUTHENTICATED_ANONYMOUSLY. This is not used in our setup

We can see authorization in working when we enable higher logging level – simply add to log4j properties:

log4j.logger.org.springframework.security=DEBUG, console

When we try to access protected page as anonymous user, among other messages we can see following entries, with AccessDeniedException thrown at the end. RoleVoter returns ACCESS_DENIED and AuthenticatedVoter returns ACCESS_ABSTAIN.

 DEBUG | 2011-07-10 12:40:30,473 | AffirmativeBased.java | decide | 53 | Voter: org.springframework.security.access.vote.RoleVoter@1ae73783, returned: -1
 DEBUG | 2011-07-10 12:40:30,474 | AffirmativeBased.java | decide | 53 | Voter: org.springframework.security.access.vote.AuthenticatedVoter@41ed8741, returned: 0
 DEBUG | 2011-07-10 12:40:30,478 | ExceptionTranslationFilter.java | handleException | 153 | Access is denied (user is anonymous); redirecting to authentication entry point
org.springframework.security.access.AccessDeniedException: Access is denied

Compare it with the log after we provide correct credentials. This time RoleVoter returns ACCESS_GRANTED and as soon as it happens, AffirmativeBased access decision manager stops querying voters and allows for access:

  DEBUG | 2011-07-10 12:43:33,468 | AffirmativeBased.java | decide | 53 | Voter: org.springframework.security.access.vote.RoleVoter@1ae73783, returned: 1
 DEBUG | 2011-07-10 12:43:33,469 | AbstractSecurityInterceptor.java | beforeInvocation | 213 | Authorization successful

There are other implementations of AccessDecisionManagers and AccessDecisionVoters but I think we will leave them for another time. Stay tuned!

Here is a full source code for the example above. Run it with:

mvn jetty:run