When mysqldump is created with --databases or --all-databases option, it will add "CREATE TABLE" and "USE" statements.
If you need to restore the dump into different database (different name) you need to remove those statements. For small files it's not a problem - just edit the dump with your favourite text editor.
However huge MySQL dumps (e.g. gigabytes big) may be a challenge. Editing with vim means loading the wohle file into the memory - not a good idea.
Solutions:
Filter dump file with grep to ignore CREATE and USE lines on the fly, while restoring:
Edit the file by overwriting CREATE and USE lines with spaces (so file size will not change).
In hexedit use TAB to switch to ASCII editing, F2 to save, F10 to exit.
% git clone https://github.com/tmuras/mysqldump-filter
% cd mysqldump-filter
% ./mysqldump-edit.php -n big_mysql_dump.sql
String to overwrite at position 839:
-->CREATE DATABASE /*!32312 IF NOT EXISTS*/ `phpmyadmin` /*!40100 DEFAULT CHARACTER SET latin1 */;<--
String to overwrite at position 936:
-->USE `phpmyadmin`;<--
% ./mysqldump-edit.php big_mysql_dump.sql
Overwriten line at position 839:
-->CREATE DATABASE /*!32312 IF NOT EXISTS*/ `phpmyadmin` /*!40100 DEFAULT CHARACTER SET latin1 */;<--
Overwriten line at position 936:
-->USE `phpmyadmin`;<--
Sometimes we want to find out what has changed in the database when some application action was made. One very simple way to do that is to use MySQL data dump with "--skip-opt" option + diff. Of course if your database is very big, it may take a while.
"--skip-opt" option will cause mysqldump to write one line with INSERT for each record - something we will need for diff to work well.
% diff snapshot1.sql snapshot2.sql
3957a3958
> INSERT INTO `mdl_config_log` VALUES (1014,2,1477218080,'assign','duedate','1209600','604800');
4718c4719
< INSERT INTO `mdl_config_plugins` VALUES (814,'assign','duedate','604800');
---
> INSERT INTO `mdl_config_plugins` VALUES (814,'assign','duedate','1209600');
13367c13368
< INSERT INTO `mdl_sessions` VALUES (4,0,'ob7ot4ngt6nddto7p5snvq3qb3',2,NULL,1477217567,1477217568,'127.0.0.1','127.0.0.1');
---
> INSERT INTO `mdl_sessions` VALUES (4,0,'ob7ot4ngt6nddto7p5snvq3qb3',2,NULL,1477217567,1477218080,'127.0.0.1','127.0.0.1');
15451c15452
< INSERT INTO `mdl_user` VALUES (2,'manual',1,0,0,0,1,'admin','$2y$10$p3IwPwFK9QAJumwKLj9mRe1MxaC3I1OUC1APcyCChcUFMMZIAmqrG','','Admin','User','nexor1984@gmail.com',0,'','','','','','','','','','','','','en','gregorian','','99',1476995496,1477217567,1476995496,1477217567,'127.0.0.1','',0,'','',1,1,0,1,1,0,0,1476995516,0,NULL,'','','','');
---
> INSERT INTO `mdl_user` VALUES (2,'manual',1,0,0,0,1,'admin','$2y$10$p3IwPwFK9QAJumwKLj9mRe1MxaC3I1OUC1APcyCChcUFMMZIAmqrG','','Admin','User','nexor1984@gmail.com',0,'','','','','','','','','','','','','en','gregorian','','99',1476995496,1477218080,1476995496,1477217567,'127.0.0.1','',0,'','',1,1,0,1,1,0,0,1476995516,0,NULL,'','','','');
16508c16509
< -- Dump completed on 2016-10-23 12:19:07
---
> -- Dump completed on 2016-10-23 12:21:59
We can see straight away that:
new record has been inserted into mdl_config_log
the value in mdl_config_plugins has changed from 604800 to 1209600
timestamps in mdl_sessions and mdl_user has been updated
Let's make another test - let's see what happens when we enable mobile web services in Moodle (I did that using Moodle UI between first and second snapshot).
% mysqldump -u root -p --skip-opt moodle31 > snapshot1.sql
% mysqldump -u root -p --skip-opt moodle31 > snapshot2.sql
% diff snapshot1.sql snapshot2.sql
2492c2492
< INSERT INTO `mdl_config` VALUES (35,'enablewebservices','0');
---
> INSERT INTO `mdl_config` VALUES (35,'enablewebservices','1');
2901c2901
< INSERT INTO `mdl_config` VALUES (447,'enablemobilewebservice','0');
---
> INSERT INTO `mdl_config` VALUES (447,'enablemobilewebservice','1');
2918a2919
> INSERT INTO `mdl_config` VALUES (465,'webserviceprotocols','rest');
3955a3957
> INSERT INTO `mdl_config_log` VALUES (1013,2,1476995587,NULL,'enablemobilewebservice','1','0');
6446c6448
< INSERT INTO `mdl_external_services` VALUES (1,'Moodle mobile web service',0,NULL,0,'moodle',1476995420,1476995523,'moodle_mobile_app',1,1);
---
> INSERT INTO `mdl_external_services` VALUES (1,'Moodle mobile web service',1,NULL,0,'moodle',1476995420,1476995598,'moodle_mobile_app',1,1);
12876a12879
> INSERT INTO `mdl_role_capabilities` VALUES (1234,1,7,'webservice/rest:use',1,1476995587,2);
13359c13362
< INSERT INTO `mdl_sessions` VALUES (2,0,'t0ivjdjkmoo24jfar7ovrqbq31',2,NULL,1476995496,1476995566,'127.0.0.1','127.0.0.1');
---
> INSERT INTO `mdl_sessions` VALUES (2,0,'t0ivjdjkmoo24jfar7ovrqbq31',2,NULL,1476995496,1476995587,'127.0.0.1','127.0.0.1');
16499c16502
< -- Dump completed on 2016-10-20 22:33:01
---
> -- Dump completed on 2016-10-20 22:33:22
order allow,deny
SetEnvIF X-Forwarded-For "89.101.211" AllowIP
Allow from env=AllowIP
“88.100.211″ is the pattern meaning “88.100.211*”
apache’s mod_setenvif must be enabled
“order allow,deny” denies all except “Allow” rules
usual “Allow from ip” rules will not work as the load balancer rewrites the “host” header, all requests are coming from the load balancer’s IP
the load balancer should set the X-Forwarded-For header
Purpose of this blog is to show how to integrate Spring Security framework with REST web services implemented using Apache CXF. To authenticate user within Spring Security framework external authentication mechanism based on token is used. The idea is to authenticate Spring Security application against external application – in our case Alfresco.
Dependencies
In order to get required libraries the following ivy script was used:
Configuration of WEB-INF/web.xml file is presented below. It contains configuration of the following frameworks:
Spring
Spring Security
Apache CXF
Hibernate
<?xml version="1.0" encoding="UTF-8"?><web-appxmlns="http://java.sun.com/xml/ns/javaee"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"version="2.5"><!-- Define configuration file for Spring --><context-param><description>Spring configuration file</description><param-name>contextConfigLocation</param-name><param-value>/WEB-INF/classes/applicationContext.xml</param-value></context-param><!-- Define configuration file for log4j --><context-param><param-name>log4jConfigLocation</param-name><param-value>/WEB-INF/classes/log4j.properties</param-value></context-param><!-- Add log4j logging --><listener><listener-class>org.springframework.web.util.Log4jConfigListener</listener-class></listener><!-- Enable Spring support --><listener><description>Spring Loader</description><listener-class>org.springframework.web.context.ContextLoaderListener</listener-class></listener><!--Define REST web services servlet --><servlet><servlet-name>CXFServlet</servlet-name><servlet-class>org.apache.cxf.transport.servlet.CXFServlet</servlet-class><load-on-startup>1</load-on-startup></servlet><servlet-mapping><servlet-name>CXFServlet</servlet-name><url-pattern>/service/*</url-pattern></servlet-mapping><!-- Hibernate filter to support opening session to get additional options --><filter><filter-name>openSessionInViewFilter</filter-name><filter-class>org.springframework.orm.hibernate4.support.OpenSessionInViewFilter</filter-class></filter><filter-mapping><filter-name>openSessionInViewFilter</filter-name><url-pattern>/service/*</url-pattern></filter-mapping><!-- Spring Security --><filter><filter-name>springSecurityFilterChain</filter-name><filter-class>org.springframework.web.filter.DelegatingFilterProxy</filter-class></filter><filter-mapping><filter-name>springSecurityFilterChain</filter-name><url-pattern>/service/*</url-pattern></filter-mapping></web-app>
We are going to put all the files related to REST web service in path /service/*, therefore in the configuration file above this path is set as url-pattern for Apache CXF Service filter (CXFServlet). Spring Security filter (springSecurityFilterChain) has the same url-pattern set in order to secure REST web services.
Spring configuration file
As defined in WEB-INF/web.xml file Spring configuration file is in the following location ‘/WEB-INF/classes/applicationContext.xml’ and is presented below. It includes files with configuration of beans required by Spring Security (resources/context/security.xml) and Apache CXF REST web service (resources/context/service.xml). Section with namespace ‘jaxrs’ corresponds to configuration of Apache CXF REST web service and section with namespace ‘security’ corresponds to configuration related to Spring Security. They will be described in more detail in the following sections.
<?xml version="1.0" encoding="UTF-8"?><beansxmlns="http://www.springframework.org/schema/beans"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xmlns:context="http://www.springframework.org/schema/context"xmlns:jaxrs="http://cxf.apache.org/jaxrs"xmlns:cxf="http://cxf.apache.org/core"xmlns:security="http://www.springframework.org/schema/security"xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd http://cxf.apache.org/jaxrs http://cxf.apache.org/schemas/jaxrs.xsd http://cxf.apache.org/core http://cxf.apache.org/schemas/core.xsd http://www.springframework.org/schema/security http://www.springframework.org/schema/security/spring-security.xsd"><!-- Enable use of annotations --><context:annotation-config/><!-- Import files with Spring bean definitions --><importresource="classpath:resources/context/service.xml"/><importresource="classpath:resources/context/security.xml"/><!-- Define REST services --><jaxrs:serverid="userService"address="/"><jaxrs:serviceBeans><refbean="userServiceBean"/></jaxrs:serviceBeans><jaxrs:extensionMappings><entrykey="xml"value="application/xml"/><entrykey="json"value="application/json"/><entrykey="feed"value="application/atom+xml"/><entrykey="html"value="text/html"/></jaxrs:extensionMappings><jaxrs:providers><beanclass="org.codehaus.jackson.jaxrs.JacksonJaxbJsonProvider"/><beanclass="org.codehaus.jackson.jaxrs.JacksonJsonProvider"/></jaxrs:providers><jaxrs:features><cxf:logging/></jaxrs:features></jaxrs:server><security:authentication-manageralias="authenticationManager"><security:authentication-providerref="myAuthenticationProvider"/></security:authentication-manager><security:global-method-securityauthentication-manager-ref="authenticationManager"jsr250-annotations="enabled"/><security:httppattern="/service/initialize/**"security="none"/><security:httpauto-config="false"entry-point-ref="http403ForbiddenEntryPoint"use-expressions="true"><security:intercept-urlpattern="/service/**"access="isAuthenticated()"/><security:custom-filterposition="FORM_LOGIN_FILTER"ref="myAuthenticationProcessingFilter"/></security:http></beans>
Apache CXF REST web service configuration
As defined in ‘/WEB-INF/classes/applicationContext.xml’ (jaxrs:server) REST service consists of one service – userService. Corresponding Spring bean (jaxrs:serviceBeans) is called ‘userServiceBean’ and is defined in /WEB-INF/classes/resources/context/service.xml file as follows:
In addition the configuration defines extension mappings (jaxrs:extensionMappings), enables logging (cxf:logging), and selects Jackson JSON-processor for processing JSON data (jaxrs:providers).
UserServiceImpl class is presented below. It implements UserService interface, produces JSON content, and path to the service is /user. It also autowires session factory, which is hibernate session factory, and defines one method retrieveProperties(). retrieveProperties() method can be accessed using the following URL: /service/user/retrieveProperties. It responds to GET request and returns JSON (determined by @Produces annotation of UserServiceImpl class) with list of property names. The URL depends on configuration of url-pattern in Apache CXF filter (web.xml) and @Path annotations.
package com.jmuras.service.user;import org.apache.commons.logging.Log;import org.apache.commons.logging.LogFactory;import org.hibernate.Query;import org.hibernate.Session;import org.hibernate.SessionFactory;import org.springframework.beans.factory.annotation.Autowired;import org.springframework.transaction.annotation.Transactional;import javax.ws.rs.*;import javax.ws.rs.core.MediaType;import javax.xml.bind.annotation.XmlElement;import javax.xml.bind.annotation.XmlElementWrapper;import java.util.ArrayList;import java.util.Collection;import java.util.List;/** * @author Joanna Muras * @date 12/12/12 * Class that implements user service methods. */@Path("/user")@Produces(MediaType.APPLICATION_JSON + "; " + MediaType.CHARSET_PARAMETER +"=UTF-8")public class UserServiceImpl implements UserService { private static Log logger = LogFactory.getLog(UserServiceImpl.class); @Autowired private SessionFactory sessionFactory; @GET @Path("/retrieveProperties") @Transactional public Collection<String> retrieveProperties() { if (logger.isDebugEnabled()) { logger.debug("Retrieve user properties"); } Session session = sessionFactory.getCurrentSession(); Query query = session.getNamedQuery("User.findProperties"); return query.list(); }}
Spring Security related configuration
As mentioned in previous section Spring security configuration is located in ‘/WEB-INF/classes/resources/context/security.xml’. Let’s assume the following scenario of authentication in Spring Security framework. There is external authentication service that we use to authorise user within Spring Security framework. When request to /service/user/retrieveProperties is made it also contains a ticket passed as parameter in URL, i.e., /service/user/retrieveProperties?ticket=.... The ticket contains all the information necessary to authenticate user in external service and see whether the ticket is still valid. External service returns all the information required by Spring Security framework to authenticate the user. The related configuration is presented below and will be discussed in the following part of this section.
<?xml version="1.0" encoding="UTF-8"?><beansxmlns="http://www.springframework.org/schema/beans"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd"><!-- My Authentication Provider --><beanid="myAuthenticationProvider"class="com.jmuras.security.AuthenticationProvider"/><beanid="myAuthenticationProcessingFilter"class="com.jmuras.security.AuthenticationProcessingFilter"><constructor-argvalue="/"/><propertyname="authenticationManager"ref="authenticationManager"/><propertyname="sessionAuthenticationStrategy"ref="sas"/>
...
</bean><beanid="http403ForbiddenEntryPoint"class="org.springframework.security.web.authentication.Http403ForbiddenEntryPoint"/><beanid="sas"class="org.springframework.security.web.authentication.session.SessionFixationProtectionStrategy"/></beans>
myAuthenticationProcessingFilter bean – AuthenticationProcessingFilter class
Responsibility of myAuthenticationProcessingFilter bean is to take ticket from the request and issue request to external authentication service to get user details. The user details are then translated to the roles defined in Spring Security annotations. AuthenticationProcessingFilter class extends AbstractAuthenticationProcessingFilter class which demands implementation of Authentication attemptAuthentication(HttpServletRequest request, HttpServletResponse response) method. This method performs actual authentication – in our example obtains user details from external authentication service in the basis of the ticket.
attemptAuthentication method is invoked only when boolean requiresAuthentication(HttpServletRequest request, HttpServletResponse response) method returns ‘true’. Implementation of requiresAuthentication in AbstractAuthenticationProcessingFilter class matches filterProcessesUrl (given in constructor of AuthenticationProcessingFilter object) against request URL, because we do not want to define access to REST web service methods on the basis of their URLs, we have to overwrite this behaviour. We assume that if there is no authentication object in context then requiresAuthentication method should return true. When authentication object is obtained getAuthenticationManager().authenticate(authentication) is invoked in the code to authenticate the user using authentication provider (AuthenticationProvider).
successfulAuthentication method is invoked when credentials are retrieved from third party authentication system and attemptAuthentication does not throw AuthenticationException exception. The method saves authentication object in security context and invokes next filter from the chain.
Implementation of myAuthenticationProcessingFilter bean, which checks ticket in Alfresco is presented below. It is assumed that there will be two additional parameters passed to request URL – alf_ticket, workgroup – /service/user/retrieveProperties?alf_ticket=...&workgroup=.... There are the following roles defined in the system:
ROLE_ADMINISTRATOR
ROLE_WORKGROUP_ADMINISTRATOR
ROLE_USER
Depending on the role user should be able to run only allowed web services.
public class AuthenticationProcessingFilter extends AbstractAuthenticationProcessingFilter { private static Log logger = LogFactory.getLog(AuthenticationProcessingFilter.class); public static final String TICKET = "alf_ticket"; public static final String WORKGROUP = "workgroup"; private String userPreferencesURL; private int retries; protected AuthenticationProcessingFilter(String defaultFilterProcessesUrl) { super(defaultFilterProcessesUrl); } public void setUserPreferencesURL(String userPreferencesURL) { this.userPreferencesURL = userPreferencesURL; } public void setRetries(int retries) { this.retries = retries; } @Override public Authentication attemptAuthentication(HttpServletRequest httpServletRequest, HttpServletResponse httpServletResponse) throws AuthenticationException, IOException, ServletException { AuthenticationToken authentication = null; String ticket = httpServletRequest.getParameter(TICKET); // Get workgroup parameter if present in URL String[] uri = httpServletRequest.getRequestURI().split("/"); String workgroup = null; if (uri != null && uri.length > 0) { int workgroupIndex = -1; for (int i = 0; i < uri.length; i++) { if (uri[i].equals(WORKGROUP)) { workgroupIndex = i; } } if (workgroupIndex > -1 && workgroupIndex + 1 < uri.length) { workgroup = uri[workgroupIndex + 1]; } } // Create an instance of HttpClient. HttpClient client = new HttpClient(); // Create user preferences request URL StringBuffer sb = new StringBuffer(userPreferencesURL); sb.append("?"); sb.append(TICKET); sb.append("="); sb.append(ticket); if (workgroup != null && workgroup.length() > 0) { sb.append("&"); sb.append(WORKGROUP); sb.append("="); sb.append(workgroup); } if (logger.isDebugEnabled()) { logger.debug("Obtain user details from Alfresco by the following request " + userPreferencesURL); } // Create a method instance. HttpMethodBase method = new GetMethod(sb.toString()); // Forward accept header method.setRequestHeader("Accept", "application/json"); // Provide custom retry handler is necessary method.getParams().setParameter(HttpMethodParams.RETRY_HANDLER, new DefaultHttpMethodRetryHandler(retries, true)); try { // Execute the method. int statusCode = client.executeMethod(method); if (statusCode == HttpStatus.SC_OK) { // Read the response body. String responseString = method.getResponseBodyAsString(); if (logger.isDebugEnabled()) { logger.debug("Following string with user details obtained from Alfresco " + userPreferencesURL); } if (responseString != null) { JSONObject json = new JSONObject(responseString); // Get authentication data from Alfresco response Collection<GrantedAuthority> authorities = new ArrayList<GrantedAuthority>(); // Depending on response obtained from Alfresco add appropriate roles to 'authorities' structure. boolean isAdmin = json.getBoolean("isAdmin"); if (isAdmin) { authorities.add(new SimpleGrantedAuthority("ROLE_ADMINISTRATOR")); } if (!json.isNull("workgroupRole")) { String workGroupRole = json.getString("workgroupRole"); if (workGroupRole != null && workGroupRole.contentEquals("SiteManager")) { authorities.add(new SimpleGrantedAuthority("ROLE_WORKGROUP_ADMINISTRATOR")); } } String principal = json.getString("userName"); // Assign user role only if user is neither workGroup administrator nor administrator if (principal != null && authorities.size() == 0) { authorities.add(new SimpleGrantedAuthority("ROLE_USER")); } // Save authorities in authentication object authentication = new AuthenticationToken(authorities); authentication.setPrincipal(principal); authentication.setCredentials(ticket); if (logger.isDebugEnabled()) { logger.debug("Authentication object filled in as follows " + authentication); } // Authenticate the user return getAuthenticationManager().authenticate(authentication); } } else { logger.error("User details could not be obtained from Alfresco " + method.getStatusLine().toString()); } } catch (HttpException e) { logger.error(e.getMessage(), e); } catch (IOException e) { logger.error(e.getMessage(), e); } catch (JSONException e) { logger.error(e.getMessage(), e); } finally { // Release the connection. method.releaseConnection(); } // Throw exception when not possible to authenticate throw new AuthenticationCredentialsNotFoundException("Authentication for ticket '" + ticket + "' unsuccessful"); } @Override protected boolean requiresAuthentication(HttpServletRequest request, HttpServletResponse response) { SecurityContext context = SecurityContextHolder.getContext(); if (context != null && context.getAuthentication() != null) { return false; } return true; } @Override protected void successfulAuthentication(HttpServletRequest request, HttpServletResponse response, FilterChain chain, Authentication authResult) throws IOException, ServletException{ SecurityContextHolder.getContext().setAuthentication(authResult); chain.doFilter(request, response); }}
myAuthenticationProvider bean – AuthenticationProvider class
This bean is responsible for returning information whether user is authenticated or not. It implements AuthenticationProvider interface, which defines 2 methods:
Authentication authenticate(Authentication authentication) – This method returns authentication object used by Spring Security for authorization. It checks data provided in authentication object passed as parameter and on that basis decides whether user should be authenticated (authentication.setAuthenticated(true)) or not. In our example if user name (authentication.getPrincipal()) was obtained from external service on the basis of the ticket, it is assumed that user is authenticated. If user is not authenticated BadCredentialsException is thrown.
boolean supports(Class aClass) – This method returns whether AuthenticationProvider object should process the request – authenticate method should be invoked. In that case only authentication object given as a parameter to authenticate method which is of AuthenticationToken type should be processed by authenticate method.
Implementation of myAuthenticationProvider bean is presented below.
public class AuthenticationProvider implements AuthenticationProvider { private static Log logger = LogFactory.getLog(AuthenticationProvider.class); @Override public Authentication authenticate(Authentication authentication) throws AuthenticationException { // Assume that if principal present user was authenticated against external service if (authentication != null && authentication.getPrincipal() != null) { authentication.setAuthenticated(true); if (logger.isDebugEnabled()) { logger.debug("User " + authentication.getPrincipal() + " authenticated"); } return authentication; } if (logger.isDebugEnabled()) { logger.debug("Ticked not valid"); } throw new BadCredentialsException("Ticked not valid"); } @Override public boolean supports(Class<?>aClass){returnaClass.equals(AuthenticationToken.class);}}
Roles definition
Roles are defined in authentication object (AuthenticationToken class). Depending on roles assigned to the user it is possible to allow the user to invoke particular web services and forbid invocation of another.The roles defined in this example are:
The roles are used in the annotations in user service (UserService) interface. The interface is implemented by UserServiceImpl class – userServiceBean bean. In the code below two methods, which are web services, are presented – retrieveOptions and createOption. The first one allows users in all three roles to invoke the web service call, and the second one allows the invocation for only workgroup administrator. The roles are the can invoke particular method are listed in @RolesAllowed annotation.
public interface UserService { public final static String TICKET = "alf_ticket"; /** * Return list of options for particular property matching criteria * @param property name of the property * @param parentProperty name of the parent property (optional) * @param parentWorkgroup name of the parent workgroup (optional) * @param parentValue parent value (optional) * @param page number of page to start * @param results number of results per page * @return list of options */ @RolesAllowed({"ROLE_USER", "ROLE_WORKGROUP_ADMINISTRATOR", "ROLE_ADMINISTRATOR"}) public Collection<OptionEntityXML> retrieveOptions(String property, Long parentId, String parentProperty, String parentWorkgroup, String parentValue, int page, int results); /** * Create new WorkGroup specific option * @param workgroup workgroup name for new option * @param wrapper values of option to be added * @return created option */ @RolesAllowed({"ROLE_WORKGROUP_ADMINISTRATOR"}) public OptionEntityXML createOption(String workgroup, OptionEntityWrapper wrapper);}
The corresponding implementation class is presented below:
@Path("/user")@Produces(MediaType.APPLICATION_JSON + "; " + MediaType.CHARSET_PARAMETER +"=UTF-8")public class UserServiceImpl implements UserService { private static Log logger = LogFactory.getLog(UserServiceImpl.class); private AlfrescoSearch search; @Autowired private SessionFactory sessionFactory; public void setSearch(AlfrescoSearch search) { this.search = search; } @GET @Path("/retrieveOptions/{property}") @Transactional @XmlElementWrapper(name = "options") @XmlElement(name = "option") public Collection<OptionEntityXML> retrieveOptions(@PathParam("property") String property, @QueryParam("parentId") Long parentId, @QueryParam("parentProperty") String parentProperty, @QueryParam("parentWorkgroup") String parentWorkgroup, @QueryParam("parentValue") String parentValue, @DefaultValue("0") @QueryParam("page") int page, @DefaultValue("0") @QueryParam("results") int results) { List<OptionEntity> list = (List<OptionEntity>) retrieveOptionsEntity(property, parentId, parentProperty, parentWorkgroup, parentValue, page, results); return getXMLList(list); } @POST @Path("/createOption") @Consumes(MediaType.APPLICATION_JSON) @Transactional public OptionEntityXML createOption(OptionEntityWrapper wrapper) { if (wrapper == null) { return null; } if (logger.isDebugEnabled()) { logger.debug("Create option with the following parameters" + wrapper.getValue() + "," + wrapper.getProperty() + "," + wrapper.getWorkgroup()); } Session session = sessionFactory.getCurrentSession(); OptionEntity entity = wrapper.getEntity(); // Remove workgroup if set wrapper.setWorkgroup(null); try { OptionEntity parentEntity = getParentOption(wrapper); if (parentEntity != null) { entity.setParent(parentEntity); } } catch (NotFoundException ex) { logger.error("Parent entity for the following parameters not found: " + wrapper.getParentId() + ", " + wrapper.getParentProperty() + ", " + wrapper.getParentWorkgroup() + ", " + wrapper.getParentValue()); } session.save(entity); return new OptionEntityXML(entity); }}
sync_binlog is a setting that (when set to 0) forces MySQL to flush a write to binary log to disk after every transaction. I have performed a quick test to see how does the setting affect the performance of MySQL and the reliability of the binary log.
Test setup:
Ubuntu Server 13.10 running in VirtualBox
MySQL 5.5.32 from the standard package, default configuration
InnoDB storage engine used
SSD storage but no battery-backed disk
I have tested the performance of a script that inserts 10000 short rows. Each test was repeated 20 times, then machine restarted and series of another 20 tests performed. Higher times for the initial tests (1-4 and 21-24) are most likely related to warm-up after reboot and should really be ignored. After each test, I’ve checked the value of the dirty (not-flushed) caches to validate the result. Note that this is pretty extreme situation – I’m emulating an application that only does writes to the database.
Time and kB of dirty cache with sync_binlog=0
Time and kB of dirty cache with sync_binlog=1
Performance
As expected, with sync_binlog=0 there are plenty of binary log changes to be flushed to disk. There is none when running with sync_binlog=1. On the other hand, the average time to perform 10000 INSERTs jumped from 23 seconds to 41 seconds – nearly doubled.
Reliability
I’ve tested how reliable is each setting by running the script with 10000 INSERTs, forcefully powering off the whole system, starting up again and then comparing records (transactions) written to the database versus those written to binary log. I have tried it few times with both setting and the same results each time:
with sync_binlog=1 the data in the database was always in sync with the binary log. That is, the last record succesfully inserted into the database, matches the last entry in binary log.
with sync_binlog=0, after reboot there were thousands of records persisted in the database but missing from the binary log.
Summary
If you care about binary log (think about master-slave replication for example) set sync_binlog to 1, if you want to squeeze a bit more performance, set it to 0.
This blog describes how to limit upload of files with size exceeding particular value to Alfresco repository.
Implementation
Restriction can be done in ContentServiceImpl class. Let’s extend this class in class CustomContentServiceImpl. We can add sizeLimit variable which is going to keep limit value in bytes and then use this limit in getWriter method.
packagecom.jmuras.alfresco.repo.content;publicclassCustomContentServiceImplextendsContentServiceImpl{...privateLongsizeLimit;publicvoidsetSizeLimit(longsizeLimit){sizeLimit=sizeLimit;}@OverridepublicContentWritergetWriter(NodeRefnodeRef,QNamepropertyQName,booleanupdate){// Get content writer from parent methodContentWritercontentWriter=super.getWriter(nodeRef,propertyQName,update);// If size limit variable is setif(contentWriterinstanceofAbstractContentWriter&&sizeLimit!=null){AbstractContentWriterabstractContentWriter=(AbstractContentWriter)contentWriter;// Set size limit for content providerContentLimitProvidercontentLimitProvider=newContentLimitProvider.SimpleFixedLimitProvider(sizeLimit);abstractContentWriter.setContentLimitProvider(contentLimitProvider);returnabstractContentWriter;}returncontentWriter;}...}
Now you have to declare new content service bean (contentService). You can do it in alfresco/patch/custom-content-services-context.xml file as follows:
<?xml version='1.0' encoding='UTF-8'?><!DOCTYPE beans PUBLIC '-//SPRING//DTD BEAN//EN' 'http://www.springframework.org/dtd/spring-beans.dtd'><beans><beanid="customBaseContentService"class="com.jmuras.alfresco.repo.content.CustomContentServiceImpl"abstract="true"parent="baseContentService"><propertyname="authenticationService"><refbean="authenticationService"/></property><propertyname="nodeService"><refbean="nodeService"/></property><!--Limit for file size in bytes--><propertyname="sizeLimit"value="2000000"/></bean><beanid="contentServiceOrig"parent="baseContentService"><propertyname="store"><refbean="fileContentStore"/></property></bean><beanid="contentService"parent="customBaseContentService"><propertyname="store"><refbean="fileContentStore"/></property></bean></beans>
Size limit variable (sizeLimit) can be also defined in alfresco-global.properties and then instead of value, e.g., 2000000 ${sizeLimit} placeholder should be used.
That change should prevent possibility of upload of files, which exceed particular size. In addition, change in Alfresco WebScript (alfresco/templates/webscripts/org/alfresco/repository/upload/upload.post.js) can be made to return appropriate error to Alfresco Share.
Existing code:
catch(e){// NOTE: Do not clean formdata temp files to allow for retries. It's possible for a temp file// to remain if max retry attempts are made, but this is rare, so leave to usual temp// file cleanup.// capture exception, annotate it accordingly and re-throwif(e.message&&e.message.indexOf("org.alfresco.service.cmr.usage.ContentQuotaException")==0){e.code=413;}else{e.code=500;e.message="Unexpected error occurred during upload of new content.";}throwe;}
New code:
catch(e){// NOTE: Do not clean formdata temp files to allow for retries. It's possible for a temp file// to remain if max retry attempts are made, but this is rare, so leave to usual temp// file cleanup.// capture exception, annotate it accordingly and re-throwif(e.message&&e.message.indexOf("org.alfresco.service.cmr.usage.ContentQuotaException")==0){e.code=413;}elseif(e.message&&e.message.indexOf("org.alfresco.repo.content.ContentLimitViolationException")==0){e.code=500;e.message="Your file exceeds upload limit.";}else{e.code=500;e.message="Unexpected error occurred during upload of new content.";}throwe;}
This blog introduces sending periodical e-mail reminders in Alfresco Activiti Workflows.
Workflow with periodical e-mail reminders
To send periodical e-mail reminders (as well as to do any other periodical action) we have to start from defining the workflow. Let’s assume that we extend already existing workflow ‘Review And Approve Activiti Process’ defined in file ‘review.bpmn20.xml’. We will add periodical reminder (‘E-mail Reminder’) to review task to remind user that this task is pending by sending him e-mail.
Corresponding XML looks like follows. Two tasks were added to it:
Timer, which is boundaryEvent with id ‘timer_review’
E-mail Reminder, which is serviceTask with id ‘mailtask_review’
<?xml version="1.0" encoding="UTF-8"?><definitionsxmlns="http://www.omg.org/spec/BPMN/20100524/MODEL"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xmlns:activiti="http://activiti.org/bpmn"xmlns:bpmndi="http://www.omg.org/spec/BPMN/20100524/DI"xmlns:omgdc="http://www.omg.org/spec/DD/20100524/DC"xmlns:omgdi="http://www.omg.org/spec/DD/20100524/DI"typeLanguage="http://www.w3.org/2001/XMLSchema"expressionLanguage="http://www.w3.org/1999/XPath"targetNamespace="http://alfresco.org"><processid="activitiReview"name="Review And Approve Activiti Process"isExecutable="true"><startEventid="start"activiti:formKey="wf:submitReviewTask"></startEvent><sequenceFlowid="flow1"sourceRef="start"targetRef="reviewTask"></sequenceFlow><userTaskid="reviewTask"name="Review Task"activiti:assignee="${bpm_assignee.properties.userName}"activiti:formKey="wf:activitiReviewTask"><extensionElements><activiti:taskListenerevent="create"class="org.alfresco.repo.workflow.activiti.tasklistener.ScriptTaskListener"><activiti:fieldname="script"><activiti:string>
if (typeof bpm_workflowDueDate != 'undefined') task.dueDate = bpm_workflowDueDate
if (typeof bpm_workflowPriority != 'undefined') task.priority = bpm_workflowPriority;
</activiti:string></activiti:field></activiti:taskListener><activiti:taskListenerevent="complete"class="org.alfresco.repo.workflow.activiti.tasklistener.ScriptTaskListener"><activiti:fieldname="script"><activiti:string>
execution.setVariable('wf_reviewOutcome', task.getVariable('wf_reviewOutcome'));
</activiti:string></activiti:field></activiti:taskListener></extensionElements></userTask><sequenceFlowid="flow2"sourceRef="reviewTask"targetRef="reviewDecision"></sequenceFlow><exclusiveGatewayid="reviewDecision"name="Review Decision"></exclusiveGateway><sequenceFlowid="flow3"sourceRef="reviewDecision"targetRef="approved"><conditionExpressionxsi:type="tFormalExpression"><![CDATA[${wf_reviewOutcome == 'Approve'}]]></conditionExpression></sequenceFlow><sequenceFlowid="flow4"sourceRef="reviewDecision"targetRef="rejected"></sequenceFlow><userTaskid="approved"name="Document Approved"activiti:assignee="${initiator.exists() ? initiator.properties.userName : 'admin'}"activiti:formKey="wf:approvedTask"><documentation>The document was reviewed and approved.</documentation><extensionElements><activiti:taskListenerevent="create"class="org.alfresco.repo.workflow.activiti.tasklistener.ScriptTaskListener"><activiti:fieldname="script"><activiti:string>
if (typeof bpm_workflowDueDate != 'undefined') task.dueDate = bpm_workflowDueDate
if (typeof bpm_workflowPriority != 'undefined') task.priority = bpm_workflowPriority;
</activiti:string></activiti:field></activiti:taskListener></extensionElements></userTask><userTaskid="rejected"name="Document Rejected"activiti:assignee="${initiator.exists() ? initiator.properties.userName : 'admin'}"activiti:formKey="wf:rejectedTask"><documentation>The document was reviewed and rejected.</documentation><extensionElements><activiti:taskListenerevent="create"class="org.alfresco.repo.workflow.activiti.tasklistener.ScriptTaskListener"><activiti:fieldname="script"><activiti:string>
if (typeof bpm_workflowDueDate != 'undefined') task.dueDate = bpm_workflowDueDate
if (typeof bpm_workflowPriority != 'undefined') task.priority = bpm_workflowPriority;
</activiti:string></activiti:field></activiti:taskListener></extensionElements></userTask><sequenceFlowid="flow5"sourceRef="approved"targetRef="end"></sequenceFlow><sequenceFlowid="flow6"sourceRef="rejected"targetRef="end"></sequenceFlow><endEventid="end"></endEvent><boundaryEventid="timer_review"name="Timer"attachedToRef="reviewTask"cancelActivity="false"><timerEventDefinition><timeCycle>0 0 0 ? * MON</timeCycle></timerEventDefinition></boundaryEvent><serviceTaskid="mailtask_review"name="E-mail Reminder"activiti:class="org.alfresco.repo.workflow.activiti.script.AlfrescoScriptDelegate"><extensionElements><activiti:fieldname="script"><activiti:string>
logger.log("E-mail reminder - review task");
// Send e-mail reminder if assignee defined
if (bpm_assignee != null) {
var mail = actions.create("mail");
mail.parameters.to = bpm_assignee.properties.userName;
mail.parameters.subject = "Task reminder - document " + bpm_workflowDescription + " pending review";
mail.parameters.ignore_send_failure = true;
mail.execute(companyhome);
}
</activiti:string></activiti:field></extensionElements></serviceTask><sequenceFlowid="flow7"sourceRef="timer_review"targetRef="mailtask_review"></sequenceFlow></process><bpmndi:BPMNDiagramid="BPMNDiagram_activitiReview"><bpmndi:BPMNPlanebpmnElement="activitiReview"id="BPMNPlane_activitiReview"><bpmndi:BPMNShapebpmnElement="start"id="BPMNShape_start"><omgdc:Boundsheight="35.0"width="35.0"x="30.0"y="200.0"></omgdc:Bounds></bpmndi:BPMNShape><bpmndi:BPMNShapebpmnElement="reviewTask"id="BPMNShape_reviewTask"><omgdc:Boundsheight="55.0"width="105.0"x="125.0"y="190.0"></omgdc:Bounds></bpmndi:BPMNShape><bpmndi:BPMNShapebpmnElement="reviewDecision"id="BPMNShape_reviewDecision"><omgdc:Boundsheight="40.0"width="40.0"x="290.0"y="197.0"></omgdc:Bounds></bpmndi:BPMNShape><bpmndi:BPMNShapebpmnElement="approved"id="BPMNShape_approved"><omgdc:Boundsheight="55.0"width="105.0"x="390.0"y="97.0"></omgdc:Bounds></bpmndi:BPMNShape><bpmndi:BPMNShapebpmnElement="rejected"id="BPMNShape_rejected"><omgdc:Boundsheight="55.0"width="105.0"x="390.0"y="297.0"></omgdc:Bounds></bpmndi:BPMNShape><bpmndi:BPMNShapebpmnElement="end"id="BPMNShape_end"><omgdc:Boundsheight="35.0"width="35.0"x="555.0"y="307.0"></omgdc:Bounds></bpmndi:BPMNShape><bpmndi:BPMNShapebpmnElement="timer_review"id="BPMNShape_timer_review"><omgdc:Boundsheight="30.0"width="30.0"x="175.0"y="230.0"></omgdc:Bounds></bpmndi:BPMNShape><bpmndi:BPMNShapebpmnElement="mailtask_review"id="BPMNShape_mailtask_review"><omgdc:Boundsheight="55.0"width="105.0"x="137.0"y="297.0"></omgdc:Bounds></bpmndi:BPMNShape><bpmndi:BPMNEdgebpmnElement="flow1"id="BPMNEdge_flow1"><omgdi:waypointx="65.0"y="217.0"></omgdi:waypoint><omgdi:waypointx="125.0"y="217.0"></omgdi:waypoint></bpmndi:BPMNEdge><bpmndi:BPMNEdgebpmnElement="flow2"id="BPMNEdge_flow2"><omgdi:waypointx="230.0"y="217.0"></omgdi:waypoint><omgdi:waypointx="290.0"y="217.0"></omgdi:waypoint></bpmndi:BPMNEdge><bpmndi:BPMNEdgebpmnElement="flow3"id="BPMNEdge_flow3"><omgdi:waypointx="310.0"y="197.0"></omgdi:waypoint><omgdi:waypointx="310.0"y="124.0"></omgdi:waypoint><omgdi:waypointx="390.0"y="124.0"></omgdi:waypoint></bpmndi:BPMNEdge><bpmndi:BPMNEdgebpmnElement="flow4"id="BPMNEdge_flow4"><omgdi:waypointx="310.0"y="237.0"></omgdi:waypoint><omgdi:waypointx="310.0"y="324.0"></omgdi:waypoint><omgdi:waypointx="390.0"y="324.0"></omgdi:waypoint></bpmndi:BPMNEdge><bpmndi:BPMNEdgebpmnElement="flow5"id="BPMNEdge_flow5"><omgdi:waypointx="495.0"y="124.0"></omgdi:waypoint><omgdi:waypointx="572.0"y="124.0"></omgdi:waypoint><omgdi:waypointx="572.0"y="307.0"></omgdi:waypoint></bpmndi:BPMNEdge><bpmndi:BPMNEdgebpmnElement="flow6"id="BPMNEdge_flow6"><omgdi:waypointx="495.0"y="324.0"></omgdi:waypoint><omgdi:waypointx="555.0"y="324.0"></omgdi:waypoint></bpmndi:BPMNEdge><bpmndi:BPMNEdgebpmnElement="flow7"id="BPMNEdge_flow7"><omgdi:waypointx="190.0"y="260.0"></omgdi:waypoint><omgdi:waypointx="189.0"y="297.0"></omgdi:waypoint></bpmndi:BPMNEdge></bpmndi:BPMNPlane></bpmndi:BPMNDiagram></definitions>
Timer
Complete reference to boundaryEvent can be found here. In our case we care about two attributes:
attachedToRef – indicates task which starts timer
cancelActivity – if set to ‘false’ does not cancel original task (defined in attachedToRef attribute)
Complete reference to timerEventDefinitions can be found here. In our case we selected to use timeCycle element, which specifies repeating interval and can be given as cron expression, e.g., run ‘E-mail Reminder’ task every Monday at 00:00am – 0 0 0 ? * MON
E-mail Reminder
This task is serviceTask, which means that it executes without any user interaction. In our case we define it as ‘org.alfresco.repo.workflow.activiti.script.AlfrescoScriptDelegate’ class to make it possible to write JavaScript code, which uses Alfresco JavaScript API. In this task, simply send e-mail with reminder to user, which was selected as review assignee.
How many lines of code does Moodle have?
The short answer is: core Moodle 2.6 has about 600k lines of code + 3rd party libraries.
Now, the long answer.
A lot of the Moodle code are 3rd party libraries that are copied mostly under “lib” directory and some other places – I have counted those separately. I have also not counted *-min.js and *-debug.js files as duplicates.
After stripping from those, here is the statistic for the core code for the latest Moodle (2.7dev build: 20131122):
class A{ private $var = 17; public function accessOtherPrivate(A $a) { echo $a->var; }}$a = new A();$b = new A();$b->accessOtherPrivate($a);
The code above works just fine, you can access private properties of different object – as long as you’re the same class. PHP manual confirms the behaviour:
Members declared as private may only be accessed by the class that defines the member.
The purpose of this blog is to show how to enable search for multi-valued properties in advanced search form in Alfresco. Let’s assume that we are looking for authors and we want to find all the documents created by author John or Mary. This feature allows to do this in single query instead of two queries.
Solution
Multi-valued search is already supported in Alfresco, but requires additional configuration. This can be done by adding additional hidden field to advanced search form. The field should follow the name format prop:{name of the filed with values separated by commas}-mode and defines logical operator that should be used to replace commas.
<configevaluator="model-type"condition="cm:content"><forms><!-- Search form --><formid="search"><field-visibility><!-- Author field--><showid="cm:author"force="true"/><!-- Field to enable multi-value author support--><showid="prop:cm:author-mode"force="true"/></field-visibility><appearance><fieldid="cm:author"set="document_additional"/><!-- Use OR logical operator for authors --><fieldid="prop:cm:author-mode"set="document_additional"><controltemplate="/org/alfresco/components/form/controls/hidden.ftl"><control-paramname="contextProperty">OR</control-param></control></field></appearance></form></forms></config>
In addition, hidden.ftl form control has to be modified to support providing field value in contextProperty parameter. The changes are presented below:
<#-- Renders a hidden form field for edit and create modes only -->
<#assign fieldValue = "">
<#if field.control.params.contextProperty??>
<#if context.properties[field.control.params.contextProperty]??>
<#assign fieldValue = context.properties[field.control.params.contextProperty]>
<#elseif args[field.control.params.contextProperty]??>
<#assign fieldValue = args[field.control.params.contextProperty]>
<#else>
<#assign fieldValue = field.control.params.contextProperty>
</#if>
<#elseif context.properties[field.name]??>
<#assign fieldValue = context.properties[field.name]>
<#else>
<#assign fieldValue = field.value>
</#if>
<#if form.mode == "edit" || form.mode == "create">
<inputtype="hidden"name="${field.name}"<#iffield.value?is_number>value="${fieldValue?c}"<#else>value="${fieldValue?html}"</#if> />
</#if>
