The latest release - 3.4 has about 800k non-comment, non-blank lines of PHP code
and about 90.7k non-comment, non-blank lines of JS code.
How did that change from version to version?
See the metrics from Moodle 2.6 to Moodle 3.4 for the PHP code:
and for the JavaScript code:
How was that calculated?
To analyze the core Moodle I have removed from it: 3-rd party libraries, "build" and "lang" directories.
I have then used cloc to calculate the number of non-comment and non-empty
lines of code.
We have a typical web application that is deployed behind a load balancer with several web servers behind.
The web servers are nearly identical - running the same OS, code, packages. But they do differ in one way - some have
a quick access to the external asset's repository. Other web servers do have the access as well but it is much slower.
It looks something like this:
Moodle context
The web application is Moodle. It's configured to use file system repository
to allow access to the assets server. One of the web servers (web1) is in a co-located network with the fast access to the storage.
Access from web2 and web3 is still possible (and configured but much slower). It would be benefictial if all the requests that trigger an access to the
external file server were routed through web1 server. Practically it means that we want to send all requests to /repository/* scripts to web1.
The solution
HAProxy can do exactly what we need:
* If a request is to http:///repository/* then route it to web1.
* Let's not compromise high availablility - in case our preferred web1 is down, send the request above to web2 or web3.
* Send all the other requests to web1 (no point in making web1 dedicated for the repository-requests only), web2 or web3.
We start HAProxy configuration with the frontend section. Custom acl rule called below "url_file" will match the requests
starting with the path /repository/.
If acl condition is met, we send the request to backend called "file_servers". Otherwise the request goes to the default
backend called "all_servers".
The definition of our backend all_servers. Nothing really unusual here - just we send a little less (16 instead of 32)
max conections to the web1 - as this one will be a bit more busy serving additional requests.
backend all_servers
server web1 10.0.3.225:80 maxconn 16 check
server web2 10.0.3.148:80 maxconn 32 check
server web3 10.0.3.50:80 maxconn 32 check
In "file_servers" backend we want to use web1 server only, unless it's down. Only then other web servers will take over - this
is done with a "backup" option of HAProxy configuration:
backend file_servers
server web1 10.0.3.225:80 maxconn 32 check
server web2 10.0.3.148:80 maxconn 32 check backup
server web3 10.0.3.50:80 maxconn 32 check backup
The configuration works as expected. The requests to http:///repository/repository_ajax.php are handled by web1. But
when web1 is shut down, web2 and web3 are being used.
This has some implications - as per documentation it
"makes sharing between processes via /tmp or /var/tmp impossible".
Basically /tmp and /var/tmp will be re-mounted for the apache process after it starts.
During service restart directories will be cleaned up and re-created.
In the main filesystem they will be visible under something like:
/tmp/systemd-private-3aa821d7448d4b21a2c379847aa20e20-apache2.service-0XM9bk
Let's see it in action - I will find any of my apache processes running:
There quite are few naming conventions for multiple-word variable names in use these days. The most popular are:
UpperCamelCase
lowerCamelCase
snake_case
There is also kebab-case convention used in LISP.
Moodle has adpoted a non-standard convention - all words are lower-case and glued together (moodlecase) - see Moodle docs. I don't think there is a name for this convention.
When mysqldump is created with --databases or --all-databases option, it will add "CREATE TABLE" and "USE" statements.
If you need to restore the dump into different database (different name) you need to remove those statements. For small files it's not a problem - just edit the dump with your favourite text editor.
However huge MySQL dumps (e.g. gigabytes big) may be a challenge. Editing with vim means loading the wohle file into the memory - not a good idea.
Solutions:
Filter dump file with grep to ignore CREATE and USE lines on the fly, while restoring:
Edit the file by overwriting CREATE and USE lines with spaces (so file size will not change).
In hexedit use TAB to switch to ASCII editing, F2 to save, F10 to exit.
% git clone https://github.com/tmuras/mysqldump-filter
% cd mysqldump-filter
% ./mysqldump-edit.php -n big_mysql_dump.sql
String to overwrite at position 839:
-->CREATE DATABASE /*!32312 IF NOT EXISTS*/ `phpmyadmin` /*!40100 DEFAULT CHARACTER SET latin1 */;<--
String to overwrite at position 936:
-->USE `phpmyadmin`;<--
% ./mysqldump-edit.php big_mysql_dump.sql
Overwriten line at position 839:
-->CREATE DATABASE /*!32312 IF NOT EXISTS*/ `phpmyadmin` /*!40100 DEFAULT CHARACTER SET latin1 */;<--
Overwriten line at position 936:
-->USE `phpmyadmin`;<--
Sometimes we want to find out what has changed in the database when some application action was made. One very simple way to do that is to use MySQL data dump with "--skip-opt" option + diff. Of course if your database is very big, it may take a while.
"--skip-opt" option will cause mysqldump to write one line with INSERT for each record - something we will need for diff to work well.
% diff snapshot1.sql snapshot2.sql
3957a3958
> INSERT INTO `mdl_config_log` VALUES (1014,2,1477218080,'assign','duedate','1209600','604800');
4718c4719
< INSERT INTO `mdl_config_plugins` VALUES (814,'assign','duedate','604800');
---
> INSERT INTO `mdl_config_plugins` VALUES (814,'assign','duedate','1209600');
13367c13368
< INSERT INTO `mdl_sessions` VALUES (4,0,'ob7ot4ngt6nddto7p5snvq3qb3',2,NULL,1477217567,1477217568,'127.0.0.1','127.0.0.1');
---
> INSERT INTO `mdl_sessions` VALUES (4,0,'ob7ot4ngt6nddto7p5snvq3qb3',2,NULL,1477217567,1477218080,'127.0.0.1','127.0.0.1');
15451c15452
< INSERT INTO `mdl_user` VALUES (2,'manual',1,0,0,0,1,'admin','$2y$10$p3IwPwFK9QAJumwKLj9mRe1MxaC3I1OUC1APcyCChcUFMMZIAmqrG','','Admin','User','nexor1984@gmail.com',0,'','','','','','','','','','','','','en','gregorian','','99',1476995496,1477217567,1476995496,1477217567,'127.0.0.1','',0,'','',1,1,0,1,1,0,0,1476995516,0,NULL,'','','','');
---
> INSERT INTO `mdl_user` VALUES (2,'manual',1,0,0,0,1,'admin','$2y$10$p3IwPwFK9QAJumwKLj9mRe1MxaC3I1OUC1APcyCChcUFMMZIAmqrG','','Admin','User','nexor1984@gmail.com',0,'','','','','','','','','','','','','en','gregorian','','99',1476995496,1477218080,1476995496,1477217567,'127.0.0.1','',0,'','',1,1,0,1,1,0,0,1476995516,0,NULL,'','','','');
16508c16509
< -- Dump completed on 2016-10-23 12:19:07
---
> -- Dump completed on 2016-10-23 12:21:59
We can see straight away that:
new record has been inserted into mdl_config_log
the value in mdl_config_plugins has changed from 604800 to 1209600
timestamps in mdl_sessions and mdl_user has been updated
Let's make another test - let's see what happens when we enable mobile web services in Moodle (I did that using Moodle UI between first and second snapshot).
% mysqldump -u root -p --skip-opt moodle31 > snapshot1.sql
% mysqldump -u root -p --skip-opt moodle31 > snapshot2.sql
% diff snapshot1.sql snapshot2.sql
2492c2492
< INSERT INTO `mdl_config` VALUES (35,'enablewebservices','0');
---
> INSERT INTO `mdl_config` VALUES (35,'enablewebservices','1');
2901c2901
< INSERT INTO `mdl_config` VALUES (447,'enablemobilewebservice','0');
---
> INSERT INTO `mdl_config` VALUES (447,'enablemobilewebservice','1');
2918a2919
> INSERT INTO `mdl_config` VALUES (465,'webserviceprotocols','rest');
3955a3957
> INSERT INTO `mdl_config_log` VALUES (1013,2,1476995587,NULL,'enablemobilewebservice','1','0');
6446c6448
< INSERT INTO `mdl_external_services` VALUES (1,'Moodle mobile web service',0,NULL,0,'moodle',1476995420,1476995523,'moodle_mobile_app',1,1);
---
> INSERT INTO `mdl_external_services` VALUES (1,'Moodle mobile web service',1,NULL,0,'moodle',1476995420,1476995598,'moodle_mobile_app',1,1);
12876a12879
> INSERT INTO `mdl_role_capabilities` VALUES (1234,1,7,'webservice/rest:use',1,1476995587,2);
13359c13362
< INSERT INTO `mdl_sessions` VALUES (2,0,'t0ivjdjkmoo24jfar7ovrqbq31',2,NULL,1476995496,1476995566,'127.0.0.1','127.0.0.1');
---
> INSERT INTO `mdl_sessions` VALUES (2,0,'t0ivjdjkmoo24jfar7ovrqbq31',2,NULL,1476995496,1476995587,'127.0.0.1','127.0.0.1');
16499c16502
< -- Dump completed on 2016-10-20 22:33:01
---
> -- Dump completed on 2016-10-20 22:33:22
order allow,deny
SetEnvIF X-Forwarded-For "89.101.211" AllowIP
Allow from env=AllowIP
“88.100.211″ is the pattern meaning “88.100.211*”
apache’s mod_setenvif must be enabled
“order allow,deny” denies all except “Allow” rules
usual “Allow from ip” rules will not work as the load balancer rewrites the “host” header, all requests are coming from the load balancer’s IP
the load balancer should set the X-Forwarded-For header
Purpose of this blog is to show how to integrate Spring Security framework with REST web services implemented using Apache CXF. To authenticate user within Spring Security framework external authentication mechanism based on token is used. The idea is to authenticate Spring Security application against external application – in our case Alfresco.
Dependencies
In order to get required libraries the following ivy script was used:
Configuration of WEB-INF/web.xml file is presented below. It contains configuration of the following frameworks:
Spring
Spring Security
Apache CXF
Hibernate
<?xml version="1.0" encoding="UTF-8"?><web-appxmlns="http://java.sun.com/xml/ns/javaee"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"version="2.5"><!-- Define configuration file for Spring --><context-param><description>Spring configuration file</description><param-name>contextConfigLocation</param-name><param-value>/WEB-INF/classes/applicationContext.xml</param-value></context-param><!-- Define configuration file for log4j --><context-param><param-name>log4jConfigLocation</param-name><param-value>/WEB-INF/classes/log4j.properties</param-value></context-param><!-- Add log4j logging --><listener><listener-class>org.springframework.web.util.Log4jConfigListener</listener-class></listener><!-- Enable Spring support --><listener><description>Spring Loader</description><listener-class>org.springframework.web.context.ContextLoaderListener</listener-class></listener><!--Define REST web services servlet --><servlet><servlet-name>CXFServlet</servlet-name><servlet-class>org.apache.cxf.transport.servlet.CXFServlet</servlet-class><load-on-startup>1</load-on-startup></servlet><servlet-mapping><servlet-name>CXFServlet</servlet-name><url-pattern>/service/*</url-pattern></servlet-mapping><!-- Hibernate filter to support opening session to get additional options --><filter><filter-name>openSessionInViewFilter</filter-name><filter-class>org.springframework.orm.hibernate4.support.OpenSessionInViewFilter</filter-class></filter><filter-mapping><filter-name>openSessionInViewFilter</filter-name><url-pattern>/service/*</url-pattern></filter-mapping><!-- Spring Security --><filter><filter-name>springSecurityFilterChain</filter-name><filter-class>org.springframework.web.filter.DelegatingFilterProxy</filter-class></filter><filter-mapping><filter-name>springSecurityFilterChain</filter-name><url-pattern>/service/*</url-pattern></filter-mapping></web-app>
We are going to put all the files related to REST web service in path /service/*, therefore in the configuration file above this path is set as url-pattern for Apache CXF Service filter (CXFServlet). Spring Security filter (springSecurityFilterChain) has the same url-pattern set in order to secure REST web services.
Spring configuration file
As defined in WEB-INF/web.xml file Spring configuration file is in the following location ‘/WEB-INF/classes/applicationContext.xml’ and is presented below. It includes files with configuration of beans required by Spring Security (resources/context/security.xml) and Apache CXF REST web service (resources/context/service.xml). Section with namespace ‘jaxrs’ corresponds to configuration of Apache CXF REST web service and section with namespace ‘security’ corresponds to configuration related to Spring Security. They will be described in more detail in the following sections.
<?xml version="1.0" encoding="UTF-8"?><beansxmlns="http://www.springframework.org/schema/beans"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xmlns:context="http://www.springframework.org/schema/context"xmlns:jaxrs="http://cxf.apache.org/jaxrs"xmlns:cxf="http://cxf.apache.org/core"xmlns:security="http://www.springframework.org/schema/security"xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd http://cxf.apache.org/jaxrs http://cxf.apache.org/schemas/jaxrs.xsd http://cxf.apache.org/core http://cxf.apache.org/schemas/core.xsd http://www.springframework.org/schema/security http://www.springframework.org/schema/security/spring-security.xsd"><!-- Enable use of annotations --><context:annotation-config/><!-- Import files with Spring bean definitions --><importresource="classpath:resources/context/service.xml"/><importresource="classpath:resources/context/security.xml"/><!-- Define REST services --><jaxrs:serverid="userService"address="/"><jaxrs:serviceBeans><refbean="userServiceBean"/></jaxrs:serviceBeans><jaxrs:extensionMappings><entrykey="xml"value="application/xml"/><entrykey="json"value="application/json"/><entrykey="feed"value="application/atom+xml"/><entrykey="html"value="text/html"/></jaxrs:extensionMappings><jaxrs:providers><beanclass="org.codehaus.jackson.jaxrs.JacksonJaxbJsonProvider"/><beanclass="org.codehaus.jackson.jaxrs.JacksonJsonProvider"/></jaxrs:providers><jaxrs:features><cxf:logging/></jaxrs:features></jaxrs:server><security:authentication-manageralias="authenticationManager"><security:authentication-providerref="myAuthenticationProvider"/></security:authentication-manager><security:global-method-securityauthentication-manager-ref="authenticationManager"jsr250-annotations="enabled"/><security:httppattern="/service/initialize/**"security="none"/><security:httpauto-config="false"entry-point-ref="http403ForbiddenEntryPoint"use-expressions="true"><security:intercept-urlpattern="/service/**"access="isAuthenticated()"/><security:custom-filterposition="FORM_LOGIN_FILTER"ref="myAuthenticationProcessingFilter"/></security:http></beans>
Apache CXF REST web service configuration
As defined in ‘/WEB-INF/classes/applicationContext.xml’ (jaxrs:server) REST service consists of one service – userService. Corresponding Spring bean (jaxrs:serviceBeans) is called ‘userServiceBean’ and is defined in /WEB-INF/classes/resources/context/service.xml file as follows:
In addition the configuration defines extension mappings (jaxrs:extensionMappings), enables logging (cxf:logging), and selects Jackson JSON-processor for processing JSON data (jaxrs:providers).
UserServiceImpl class is presented below. It implements UserService interface, produces JSON content, and path to the service is /user. It also autowires session factory, which is hibernate session factory, and defines one method retrieveProperties(). retrieveProperties() method can be accessed using the following URL: /service/user/retrieveProperties. It responds to GET request and returns JSON (determined by @Produces annotation of UserServiceImpl class) with list of property names. The URL depends on configuration of url-pattern in Apache CXF filter (web.xml) and @Path annotations.
package com.jmuras.service.user;import org.apache.commons.logging.Log;import org.apache.commons.logging.LogFactory;import org.hibernate.Query;import org.hibernate.Session;import org.hibernate.SessionFactory;import org.springframework.beans.factory.annotation.Autowired;import org.springframework.transaction.annotation.Transactional;import javax.ws.rs.*;import javax.ws.rs.core.MediaType;import javax.xml.bind.annotation.XmlElement;import javax.xml.bind.annotation.XmlElementWrapper;import java.util.ArrayList;import java.util.Collection;import java.util.List;/** * @author Joanna Muras * @date 12/12/12 * Class that implements user service methods. */@Path("/user")@Produces(MediaType.APPLICATION_JSON + "; " + MediaType.CHARSET_PARAMETER +"=UTF-8")public class UserServiceImpl implements UserService { private static Log logger = LogFactory.getLog(UserServiceImpl.class); @Autowired private SessionFactory sessionFactory; @GET @Path("/retrieveProperties") @Transactional public Collection<String> retrieveProperties() { if (logger.isDebugEnabled()) { logger.debug("Retrieve user properties"); } Session session = sessionFactory.getCurrentSession(); Query query = session.getNamedQuery("User.findProperties"); return query.list(); }}
Spring Security related configuration
As mentioned in previous section Spring security configuration is located in ‘/WEB-INF/classes/resources/context/security.xml’. Let’s assume the following scenario of authentication in Spring Security framework. There is external authentication service that we use to authorise user within Spring Security framework. When request to /service/user/retrieveProperties is made it also contains a ticket passed as parameter in URL, i.e., /service/user/retrieveProperties?ticket=.... The ticket contains all the information necessary to authenticate user in external service and see whether the ticket is still valid. External service returns all the information required by Spring Security framework to authenticate the user. The related configuration is presented below and will be discussed in the following part of this section.
<?xml version="1.0" encoding="UTF-8"?><beansxmlns="http://www.springframework.org/schema/beans"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd"><!-- My Authentication Provider --><beanid="myAuthenticationProvider"class="com.jmuras.security.AuthenticationProvider"/><beanid="myAuthenticationProcessingFilter"class="com.jmuras.security.AuthenticationProcessingFilter"><constructor-argvalue="/"/><propertyname="authenticationManager"ref="authenticationManager"/><propertyname="sessionAuthenticationStrategy"ref="sas"/>
...
</bean><beanid="http403ForbiddenEntryPoint"class="org.springframework.security.web.authentication.Http403ForbiddenEntryPoint"/><beanid="sas"class="org.springframework.security.web.authentication.session.SessionFixationProtectionStrategy"/></beans>
myAuthenticationProcessingFilter bean – AuthenticationProcessingFilter class
Responsibility of myAuthenticationProcessingFilter bean is to take ticket from the request and issue request to external authentication service to get user details. The user details are then translated to the roles defined in Spring Security annotations. AuthenticationProcessingFilter class extends AbstractAuthenticationProcessingFilter class which demands implementation of Authentication attemptAuthentication(HttpServletRequest request, HttpServletResponse response) method. This method performs actual authentication – in our example obtains user details from external authentication service in the basis of the ticket.
attemptAuthentication method is invoked only when boolean requiresAuthentication(HttpServletRequest request, HttpServletResponse response) method returns ‘true’. Implementation of requiresAuthentication in AbstractAuthenticationProcessingFilter class matches filterProcessesUrl (given in constructor of AuthenticationProcessingFilter object) against request URL, because we do not want to define access to REST web service methods on the basis of their URLs, we have to overwrite this behaviour. We assume that if there is no authentication object in context then requiresAuthentication method should return true. When authentication object is obtained getAuthenticationManager().authenticate(authentication) is invoked in the code to authenticate the user using authentication provider (AuthenticationProvider).
successfulAuthentication method is invoked when credentials are retrieved from third party authentication system and attemptAuthentication does not throw AuthenticationException exception. The method saves authentication object in security context and invokes next filter from the chain.
Implementation of myAuthenticationProcessingFilter bean, which checks ticket in Alfresco is presented below. It is assumed that there will be two additional parameters passed to request URL – alf_ticket, workgroup – /service/user/retrieveProperties?alf_ticket=...&workgroup=.... There are the following roles defined in the system:
ROLE_ADMINISTRATOR
ROLE_WORKGROUP_ADMINISTRATOR
ROLE_USER
Depending on the role user should be able to run only allowed web services.
public class AuthenticationProcessingFilter extends AbstractAuthenticationProcessingFilter { private static Log logger = LogFactory.getLog(AuthenticationProcessingFilter.class); public static final String TICKET = "alf_ticket"; public static final String WORKGROUP = "workgroup"; private String userPreferencesURL; private int retries; protected AuthenticationProcessingFilter(String defaultFilterProcessesUrl) { super(defaultFilterProcessesUrl); } public void setUserPreferencesURL(String userPreferencesURL) { this.userPreferencesURL = userPreferencesURL; } public void setRetries(int retries) { this.retries = retries; } @Override public Authentication attemptAuthentication(HttpServletRequest httpServletRequest, HttpServletResponse httpServletResponse) throws AuthenticationException, IOException, ServletException { AuthenticationToken authentication = null; String ticket = httpServletRequest.getParameter(TICKET); // Get workgroup parameter if present in URL String[] uri = httpServletRequest.getRequestURI().split("/"); String workgroup = null; if (uri != null && uri.length > 0) { int workgroupIndex = -1; for (int i = 0; i < uri.length; i++) { if (uri[i].equals(WORKGROUP)) { workgroupIndex = i; } } if (workgroupIndex > -1 && workgroupIndex + 1 < uri.length) { workgroup = uri[workgroupIndex + 1]; } } // Create an instance of HttpClient. HttpClient client = new HttpClient(); // Create user preferences request URL StringBuffer sb = new StringBuffer(userPreferencesURL); sb.append("?"); sb.append(TICKET); sb.append("="); sb.append(ticket); if (workgroup != null && workgroup.length() > 0) { sb.append("&"); sb.append(WORKGROUP); sb.append("="); sb.append(workgroup); } if (logger.isDebugEnabled()) { logger.debug("Obtain user details from Alfresco by the following request " + userPreferencesURL); } // Create a method instance. HttpMethodBase method = new GetMethod(sb.toString()); // Forward accept header method.setRequestHeader("Accept", "application/json"); // Provide custom retry handler is necessary method.getParams().setParameter(HttpMethodParams.RETRY_HANDLER, new DefaultHttpMethodRetryHandler(retries, true)); try { // Execute the method. int statusCode = client.executeMethod(method); if (statusCode == HttpStatus.SC_OK) { // Read the response body. String responseString = method.getResponseBodyAsString(); if (logger.isDebugEnabled()) { logger.debug("Following string with user details obtained from Alfresco " + userPreferencesURL); } if (responseString != null) { JSONObject json = new JSONObject(responseString); // Get authentication data from Alfresco response Collection<GrantedAuthority> authorities = new ArrayList<GrantedAuthority>(); // Depending on response obtained from Alfresco add appropriate roles to 'authorities' structure. boolean isAdmin = json.getBoolean("isAdmin"); if (isAdmin) { authorities.add(new SimpleGrantedAuthority("ROLE_ADMINISTRATOR")); } if (!json.isNull("workgroupRole")) { String workGroupRole = json.getString("workgroupRole"); if (workGroupRole != null && workGroupRole.contentEquals("SiteManager")) { authorities.add(new SimpleGrantedAuthority("ROLE_WORKGROUP_ADMINISTRATOR")); } } String principal = json.getString("userName"); // Assign user role only if user is neither workGroup administrator nor administrator if (principal != null && authorities.size() == 0) { authorities.add(new SimpleGrantedAuthority("ROLE_USER")); } // Save authorities in authentication object authentication = new AuthenticationToken(authorities); authentication.setPrincipal(principal); authentication.setCredentials(ticket); if (logger.isDebugEnabled()) { logger.debug("Authentication object filled in as follows " + authentication); } // Authenticate the user return getAuthenticationManager().authenticate(authentication); } } else { logger.error("User details could not be obtained from Alfresco " + method.getStatusLine().toString()); } } catch (HttpException e) { logger.error(e.getMessage(), e); } catch (IOException e) { logger.error(e.getMessage(), e); } catch (JSONException e) { logger.error(e.getMessage(), e); } finally { // Release the connection. method.releaseConnection(); } // Throw exception when not possible to authenticate throw new AuthenticationCredentialsNotFoundException("Authentication for ticket '" + ticket + "' unsuccessful"); } @Override protected boolean requiresAuthentication(HttpServletRequest request, HttpServletResponse response) { SecurityContext context = SecurityContextHolder.getContext(); if (context != null && context.getAuthentication() != null) { return false; } return true; } @Override protected void successfulAuthentication(HttpServletRequest request, HttpServletResponse response, FilterChain chain, Authentication authResult) throws IOException, ServletException{ SecurityContextHolder.getContext().setAuthentication(authResult); chain.doFilter(request, response); }}
myAuthenticationProvider bean – AuthenticationProvider class
This bean is responsible for returning information whether user is authenticated or not. It implements AuthenticationProvider interface, which defines 2 methods:
Authentication authenticate(Authentication authentication) – This method returns authentication object used by Spring Security for authorization. It checks data provided in authentication object passed as parameter and on that basis decides whether user should be authenticated (authentication.setAuthenticated(true)) or not. In our example if user name (authentication.getPrincipal()) was obtained from external service on the basis of the ticket, it is assumed that user is authenticated. If user is not authenticated BadCredentialsException is thrown.
boolean supports(Class aClass) – This method returns whether AuthenticationProvider object should process the request – authenticate method should be invoked. In that case only authentication object given as a parameter to authenticate method which is of AuthenticationToken type should be processed by authenticate method.
Implementation of myAuthenticationProvider bean is presented below.
public class AuthenticationProvider implements AuthenticationProvider { private static Log logger = LogFactory.getLog(AuthenticationProvider.class); @Override public Authentication authenticate(Authentication authentication) throws AuthenticationException { // Assume that if principal present user was authenticated against external service if (authentication != null && authentication.getPrincipal() != null) { authentication.setAuthenticated(true); if (logger.isDebugEnabled()) { logger.debug("User " + authentication.getPrincipal() + " authenticated"); } return authentication; } if (logger.isDebugEnabled()) { logger.debug("Ticked not valid"); } throw new BadCredentialsException("Ticked not valid"); } @Override public boolean supports(Class<?>aClass){returnaClass.equals(AuthenticationToken.class);}}
Roles definition
Roles are defined in authentication object (AuthenticationToken class). Depending on roles assigned to the user it is possible to allow the user to invoke particular web services and forbid invocation of another.The roles defined in this example are:
The roles are used in the annotations in user service (UserService) interface. The interface is implemented by UserServiceImpl class – userServiceBean bean. In the code below two methods, which are web services, are presented – retrieveOptions and createOption. The first one allows users in all three roles to invoke the web service call, and the second one allows the invocation for only workgroup administrator. The roles are the can invoke particular method are listed in @RolesAllowed annotation.
public interface UserService { public final static String TICKET = "alf_ticket"; /** * Return list of options for particular property matching criteria * @param property name of the property * @param parentProperty name of the parent property (optional) * @param parentWorkgroup name of the parent workgroup (optional) * @param parentValue parent value (optional) * @param page number of page to start * @param results number of results per page * @return list of options */ @RolesAllowed({"ROLE_USER", "ROLE_WORKGROUP_ADMINISTRATOR", "ROLE_ADMINISTRATOR"}) public Collection<OptionEntityXML> retrieveOptions(String property, Long parentId, String parentProperty, String parentWorkgroup, String parentValue, int page, int results); /** * Create new WorkGroup specific option * @param workgroup workgroup name for new option * @param wrapper values of option to be added * @return created option */ @RolesAllowed({"ROLE_WORKGROUP_ADMINISTRATOR"}) public OptionEntityXML createOption(String workgroup, OptionEntityWrapper wrapper);}
The corresponding implementation class is presented below:
@Path("/user")@Produces(MediaType.APPLICATION_JSON + "; " + MediaType.CHARSET_PARAMETER +"=UTF-8")public class UserServiceImpl implements UserService { private static Log logger = LogFactory.getLog(UserServiceImpl.class); private AlfrescoSearch search; @Autowired private SessionFactory sessionFactory; public void setSearch(AlfrescoSearch search) { this.search = search; } @GET @Path("/retrieveOptions/{property}") @Transactional @XmlElementWrapper(name = "options") @XmlElement(name = "option") public Collection<OptionEntityXML> retrieveOptions(@PathParam("property") String property, @QueryParam("parentId") Long parentId, @QueryParam("parentProperty") String parentProperty, @QueryParam("parentWorkgroup") String parentWorkgroup, @QueryParam("parentValue") String parentValue, @DefaultValue("0") @QueryParam("page") int page, @DefaultValue("0") @QueryParam("results") int results) { List<OptionEntity> list = (List<OptionEntity>) retrieveOptionsEntity(property, parentId, parentProperty, parentWorkgroup, parentValue, page, results); return getXMLList(list); } @POST @Path("/createOption") @Consumes(MediaType.APPLICATION_JSON) @Transactional public OptionEntityXML createOption(OptionEntityWrapper wrapper) { if (wrapper == null) { return null; } if (logger.isDebugEnabled()) { logger.debug("Create option with the following parameters" + wrapper.getValue() + "," + wrapper.getProperty() + "," + wrapper.getWorkgroup()); } Session session = sessionFactory.getCurrentSession(); OptionEntity entity = wrapper.getEntity(); // Remove workgroup if set wrapper.setWorkgroup(null); try { OptionEntity parentEntity = getParentOption(wrapper); if (parentEntity != null) { entity.setParent(parentEntity); } } catch (NotFoundException ex) { logger.error("Parent entity for the following parameters not found: " + wrapper.getParentId() + ", " + wrapper.getParentProperty() + ", " + wrapper.getParentWorkgroup() + ", " + wrapper.getParentValue()); } session.save(entity); return new OptionEntityXML(entity); }}
sync_binlog is a setting that (when set to 0) forces MySQL to flush a write to binary log to disk after every transaction. I have performed a quick test to see how does the setting affect the performance of MySQL and the reliability of the binary log.
Test setup:
Ubuntu Server 13.10 running in VirtualBox
MySQL 5.5.32 from the standard package, default configuration
InnoDB storage engine used
SSD storage but no battery-backed disk
I have tested the performance of a script that inserts 10000 short rows. Each test was repeated 20 times, then machine restarted and series of another 20 tests performed. Higher times for the initial tests (1-4 and 21-24) are most likely related to warm-up after reboot and should really be ignored. After each test, I’ve checked the value of the dirty (not-flushed) caches to validate the result. Note that this is pretty extreme situation – I’m emulating an application that only does writes to the database.
Time and kB of dirty cache with sync_binlog=0
Time and kB of dirty cache with sync_binlog=1
Performance
As expected, with sync_binlog=0 there are plenty of binary log changes to be flushed to disk. There is none when running with sync_binlog=1. On the other hand, the average time to perform 10000 INSERTs jumped from 23 seconds to 41 seconds – nearly doubled.
Reliability
I’ve tested how reliable is each setting by running the script with 10000 INSERTs, forcefully powering off the whole system, starting up again and then comparing records (transactions) written to the database versus those written to binary log. I have tried it few times with both setting and the same results each time:
with sync_binlog=1 the data in the database was always in sync with the binary log. That is, the last record succesfully inserted into the database, matches the last entry in binary log.
with sync_binlog=0, after reboot there were thousands of records persisted in the database but missing from the binary log.
Summary
If you care about binary log (think about master-slave replication for example) set sync_binlog to 1, if you want to squeeze a bit more performance, set it to 0.
This blog describes how to limit upload of files with size exceeding particular value to Alfresco repository.
Implementation
Restriction can be done in ContentServiceImpl class. Let’s extend this class in class CustomContentServiceImpl. We can add sizeLimit variable which is going to keep limit value in bytes and then use this limit in getWriter method.
packagecom.jmuras.alfresco.repo.content;publicclassCustomContentServiceImplextendsContentServiceImpl{...privateLongsizeLimit;publicvoidsetSizeLimit(longsizeLimit){sizeLimit=sizeLimit;}@OverridepublicContentWritergetWriter(NodeRefnodeRef,QNamepropertyQName,booleanupdate){// Get content writer from parent methodContentWritercontentWriter=super.getWriter(nodeRef,propertyQName,update);// If size limit variable is setif(contentWriterinstanceofAbstractContentWriter&&sizeLimit!=null){AbstractContentWriterabstractContentWriter=(AbstractContentWriter)contentWriter;// Set size limit for content providerContentLimitProvidercontentLimitProvider=newContentLimitProvider.SimpleFixedLimitProvider(sizeLimit);abstractContentWriter.setContentLimitProvider(contentLimitProvider);returnabstractContentWriter;}returncontentWriter;}...}
Now you have to declare new content service bean (contentService). You can do it in alfresco/patch/custom-content-services-context.xml file as follows:
<?xml version='1.0' encoding='UTF-8'?><!DOCTYPE beans PUBLIC '-//SPRING//DTD BEAN//EN' 'http://www.springframework.org/dtd/spring-beans.dtd'><beans><beanid="customBaseContentService"class="com.jmuras.alfresco.repo.content.CustomContentServiceImpl"abstract="true"parent="baseContentService"><propertyname="authenticationService"><refbean="authenticationService"/></property><propertyname="nodeService"><refbean="nodeService"/></property><!--Limit for file size in bytes--><propertyname="sizeLimit"value="2000000"/></bean><beanid="contentServiceOrig"parent="baseContentService"><propertyname="store"><refbean="fileContentStore"/></property></bean><beanid="contentService"parent="customBaseContentService"><propertyname="store"><refbean="fileContentStore"/></property></bean></beans>
Size limit variable (sizeLimit) can be also defined in alfresco-global.properties and then instead of value, e.g., 2000000 ${sizeLimit} placeholder should be used.
That change should prevent possibility of upload of files, which exceed particular size. In addition, change in Alfresco WebScript (alfresco/templates/webscripts/org/alfresco/repository/upload/upload.post.js) can be made to return appropriate error to Alfresco Share.
Existing code:
catch(e){// NOTE: Do not clean formdata temp files to allow for retries. It's possible for a temp file// to remain if max retry attempts are made, but this is rare, so leave to usual temp// file cleanup.// capture exception, annotate it accordingly and re-throwif(e.message&&e.message.indexOf("org.alfresco.service.cmr.usage.ContentQuotaException")==0){e.code=413;}else{e.code=500;e.message="Unexpected error occurred during upload of new content.";}throwe;}
New code:
catch(e){// NOTE: Do not clean formdata temp files to allow for retries. It's possible for a temp file// to remain if max retry attempts are made, but this is rare, so leave to usual temp// file cleanup.// capture exception, annotate it accordingly and re-throwif(e.message&&e.message.indexOf("org.alfresco.service.cmr.usage.ContentQuotaException")==0){e.code=413;}elseif(e.message&&e.message.indexOf("org.alfresco.repo.content.ContentLimitViolationException")==0){e.code=500;e.message="Your file exceeds upload limit.";}else{e.code=500;e.message="Unexpected error occurred during upload of new content.";}throwe;}
