Programming and Computer tips

EJBs and Servlets

2009-02-13T18:53:00.000-08:00

EJBs and Servlets
Question:
Both Enterprise JavaBeans and Servlets are server side components. What are the major differences between them? Specifically, what is the difference between a Session Bean and a Servlet?

Answer:
Enterprise JavaBeans are components meant to encapsulate business logic. They do not handle presentation and have a precise set of restrictions they must obey. An EJB may not manage threads, access files with the java.io package, have a GUI, accept socket connections, or load native libraries. These restrictions are imposed because EJBs execute inside an EJB container, roughly analogous to the way servlets execute in a servlet container. Where servlets can be used to perform many arbitrary tasks, including the generation of HTML pages, EJBs are used almost entirely as business objects. A session bean represents a client for a session or transaction and an entity bean represents a persistent business object that is usually stored in a database. Unlike servlets, a single session bean may exist per client. A single servlet may serve many clients. A session bean usually mediates client access to entity beans, shielding the client from the details of managing state.

A very common model is for a servlet to act as a client of a session bean. In this multi-tier model, a user talks to a servlet or JavaServer Page through a Web browser. The servlet manages the session, keeping track of a session bean for each of its clients and forwarding transaction requests to the session bean. The session bean may then access any number of entity beans to do things like lookup customer account data. The session bean executes its business logic and returns results to the servlet which then formats a result for the Web browser.

One of the ideas behind EJBs is to componentize server-side business logic and move it closer to the database. EJBs and servlets will typically not run on the same server. A key thing to remember is that both EJBs and servlets must execute inside of a container. For servlets, this is usually a Web server interfaced with a servlet engine. For EJBs, this is an EJB server such as the reference implementation provided with the Java 2 Enterprise Edition SDK.

What to do if Remote JMS Client can not get factory and throw classcastexception?

2008-12-06T16:38:00.000-08:00

What to do if Remote JMS Client can not get factory and throw classcastexception?

First: Add the following property to initialcontext like:

props.put("java.naming.factory.initial", "org.jnp.interfaces.NamingContextFactory");
props.put("java.naming.provider.url", "jnp://localhost:1099");
props.put("java.naming.factory.url.pkgs", "org.jboss.naming:org.jnp.interfaces");

ctx = new InitialContext(props);

Second: Check the version of jar file: jboss-messaging-client. It has to have the same version with jms in jboss.

That's all

JMS on JBoss using EJB3 MDB

2008-09-25T17:14:00.000-07:00

Introduction

This simple JMS application has you submit a form which sends a message to a JBoss queue. A Message-Driven Bean (MDB) will be listening on that queue destination and receives the mssage and processes it. What makes this lesson different than this one is that this MDB example runs in an EJB3 container and the MDB is exposed using a simple annotation. To appreciate this simple example, it will really help to view the 'old way' of having to do things in the link above. With the EJB3 approach there is a lot less configuration. We don not need to configure the jboss-web.xml, destinations-service.xml, ejb-jar.xml, or the jboss.xml.

The Setup

Download the source Download the source above and unzip it into your project directory of choice. For eclipse users the source comes with an eclipse project so you can just import that project file and be good to go. To build using ant, type 'ant all' or if you want to build and deploy to jboss type 'ant deploy' to build and deploy the ear to JBoss.

Java5 The Java5 JDK needs to be set as your environment. JAVA_HOME env variable needs to be defined pointing to yth the JDK home directory.

JBoss 4.2+ - Download JBoss 4.2 here http://labs.jboss.com/portal/jbossas/download. (This should also run on 4.0.5 if it's patched to support EJB3 using the JEMS installer.)

Running the example If you've built the project usingg ant (ant deploy), the ear will be deployed to jboss and you simply go to your jboss home dir/bin and start jboss with run.sh (Nix flavors) or run.bat (Windows). If you want to just see the app run you can download the ear above and put the ear into your jboss/default/deploy directory. Once jboss is started, go the following url: http://localhost:8080/rr-jms

LongProcessMessageBean

This is our Message Driven Bean. Notice the EJB3 annotations. The annotation tells the container to deploy this as an Message Driven Bean and it also creates the queue based off our annotation. We are also using Resource injection with the @Resource annotation which injects our MessageDrivenContext message which we use if their was some business logic problem and we need to do a rollback in the catch block ( context.setRollbackOnly() .) All this MDB does is mimic calling some business process (doLongProcess). You'll see in your jboss console the println statements when this gets called. (Of course in real life you'd be using Logging and not stupid println statements.)


import javax.annotation.Resource;
import javax.ejb.ActivationConfigProperty;
import javax.ejb.MessageDriven;
import javax.ejb.MessageDrivenContext;
import javax.jms.Message;
import javax.jms.MessageListener;
import javax.jms.ObjectMessage;

import net.learntechnology.dto.ProcessDTO;
import net.learntechnology.util.LongProcessService;
import net.learntechnology.util.ProcessResult;

@MessageDriven(name="LongProcessMessageBean", activationConfig = {
   @ActivationConfigProperty(propertyName="destinationType", propertyValue="javax.jms.Queue"),
   @ActivationConfigProperty(propertyName="destination", propertyValue="queue/examples/OurSampleQueue")

})
public class LongProcessMessageBean implements MessageListener {

   @Resource
   private MessageDrivenContext context;

   public void onMessage(Message message) {
       String name = null;
       try {
           if (message instanceof ObjectMessage) {
               ObjectMessage objMessage = (ObjectMessage) message;
               Object obj = objMessage.getObject();
               if (obj instanceof ProcessDTO) {
                   name = ((ProcessDTO)obj).getName();
                   System.out.println("****************************************************");
                   System.out.println("LongProcessMessageBean.onMessage(): Received message. NAME: "+name);
                   System.out.println("****************************************************");
                   System.out.println("Now calling LongProcessService.doLongProcess");

                   ProcessResult result = LongProcessService.doLongProcess((ProcessDTO)obj);
                } else {
                   System.err.println("Expecting ProcessDTO in Message");
               }
           } else {
               System.err.println("Expecting Object Message");
           }
           System.out.println("*******************************************");
           System.out.println("Leaving LongProcessMessageBean.onMessage(). NAME: "+name );
           System.out.println("*******************************************");
       } catch (Throwable t) {
           t.printStackTrace();
           context.setRollbackOnly();
       }
   }    
}

index.jsp form

A simple JSP page. All it does is submit some text to our Servlet (AppServlet).


   Your name: lt;input type="text" name="name" size="40">

AppServlet

Our AppServlet takes the request of text submitted from index.jsp and passes it to our JmsProducer. Our JmsProducer will need to know the jndi names of the connection factory and the queue name, so we are defining them here as well and passing them to our JmsProducer class.


public class AppServlet extends HttpServlet {
   private static final String QUEUE_CONNECTION_FACTORY = "ConnectionFactory";
   private static final String LONG_PROCESS_QUEUE = "queue/examples/OurSampleQueue";

   public void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
       doPost(request, response);
   }

   public void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
       System.out.println("in doPost of AppServlet..");
       String name = request.getParameter("name");
       System.out.println("name: "+name );
       ProcessDTO processDTO = new ProcessDTO();
       processDTO.setName(name);
       JmsProducer.sendMessage(processDTO, QUEUE_CONNECTION_FACTORY, LONG_PROCESS_QUEUE);
       request.setAttribute("started",true);
       RequestDispatcher dispatcher =  getServletContext().getRequestDispatcher("/index.jsp");
       dispatcher.forward(request, response);
   }
}

JmsProducer

This is our JMS client code that sends the message to the queue. We pushed the Connection lookup and the Destination lookup into another class called 'ServiceLocator' which is shown next.


public class JmsProducer {
   private JmsProducer() {}
   public static void sendMessage(Serializable payload, String connectionFactoryJndiName,
       String destinationJndiName) throws JmsProducerException {
       try {
           ConnectionFactory connectionFactory = null;
           Connection connection = null;
           Session session = null;
           Destination destination = null;
           MessageProducer messageProducer = null;
           ObjectMessage message = null;
           System.out.println("In sendMessage of JmsProducter,
               getting ConnectionFactory for jndi name: "+connectionFactoryJndiName );
           connectionFactory = ServiceLocator.getJmsConnectionFactory(
                                                    connectionFactoryJndiName);

           connection = connectionFactory.createConnection();
           session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
           destination = ServiceLocator.getJmsDestination(destinationJndiName);
           messageProducer = session.createProducer(destination);
           message = session.createObjectMessage(payload);
           messageProducer.send(message);
           System.out.println("Message sent to messageProducer");
           messageProducer.close();
           session.close();
           connection.close();
       } catch (JMSException je) {
           throw new JmsProducerException(je);
       } catch (ServiceLocatorException sle) {
           throw new JmsProducerException(sle);
       }
   }
}

ServiceLocator

This is a simple utility class used to look up our Connection and Destination via a JNDI lookup. In real life we might use a cache as well to store these lookups.


public class ServiceLocator {
  private ServiceLocator() {}
    public static ConnectionFactory getJmsConnectionFactory(String jmsConnectionFactoryJndiName)
            throws ServiceLocatorException {
       ConnectionFactory jmsConnectionFactory = null;
       try {
           Context ctx = new InitialContext();
           jmsConnectionFactory = (ConnectionFactory) ctx.lookup(jmsConnectionFactoryJndiName);
       } catch (ClassCastException cce) {
           throw new ServiceLocatorException(cce);
       } catch (NamingException ne) {
           throw new ServiceLocatorException(ne);
       }
       return jmsConnectionFactory;
   }
   public static Destination getJmsDestination(String jmsDestinationJndiName)
           throws ServiceLocatorException {
       Destination jmsDestination = null;
       try {
           Context ctx = new InitialContext();
           jmsDestination = (Destination) ctx.lookup(jmsDestinationJndiName);
       } catch (ClassCastException cce) {
           throw new ServiceLocatorException(cce);
       } catch (NamingException ne) {
           throw new ServiceLocatorException(ne);
       }
       return jmsDestination;
   }
}

You might be wondering how to set the IntialContext with the proper host name to do its lookups. The best place to do that is in a jndi.properties file (which this app uses). Our jndi.properties file looks like this:


#jndi.properties
java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory
java.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces
java.naming.provider.url=localhost:1099

application.xml

Necessary for our ear deployment to jboss.



            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
            xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee
                                http://java.sun.com/xml/ns/j2ee/application_1_4.xsd"
            version="1.4">
   rr-jms-ejb3 Example EAR
  
       
           
               rr-jms-webapp.war
               rr-jms
           
       
       
           conf.jar
       
       
           common.jar
       
  
       
           rr-jms-ejb.jar

PHP - How to parse RSS?

2008-08-03T07:16:00.000-07:00

It is very easy to parse RSS and print out all information of RSS as following
All we need is calling function simplexml_load_string


$rss = '';
$url = "http://www.dantri.com.vn/Sukien.rss";
$rss_file = file_get_contents($url);
if(!$feed = simplexml_load_string($rss_file)){
 die("Cannot load RSS Feed. This application supports RSS 1.0 and 2.0");
}

// print out the title of the feed
echo '';
echo $feed->channel->title;
echo '
';
// check for RSS Version
$items = ($feed['version'] != '') ? $feed->channel->item : $feed->item;
// PRINT OUT ITEMS
/*---------------------------------------------------------------------------------------*/
foreach($items as $item)
{
 echo $item->description.'

';
}
/*---------------------------------------------------------------------------------------*/

How to log sql statement stored in PreparedStatement

2008-05-22T21:17:00.000-07:00



Take the following program.

import java.net.URL;
import java.sql.*;

class JDBCapp  {
  static MyConnection theConn;

  public static void main (String args[]) {
    new JDBCapp().doit();
    }

  public void doit() {
    theConn = new MyConnection();
    theConn.connect("EAS Demo DB V3", "dba", "sql");

PreparedStatement prepstmt;
try {
  prepstmt = theConn.dbConn.prepareStatement
   ("SELECT emp_id FROM employee" );
  prepstmt.execute();
  prepstmt.close();
  }
catch (Exception e) { e.printStackTrace(); }
  theConn.disconnect();
  }
}


class MyConnection {
  Connection dbConn = null;
  void connect(String db, String user, String passw) {
    try {
      Driver d = 
       (Driver)Class.forName("sun.jdbc.odbc.JdbcOdbcDriver").newInstance();
      String URL = "jdbc:odbc:" + db;
      dbConn = DriverManager.getConnection(URL, user, passw);
      java.io.PrintWriter w =
        new java.io.PrintWriter
           (new java.io.OutputStreamWriter(System.out));
      DriverManager.setLogWriter(w);
      }
    catch (Exception e) {
      e.printStackTrace();
      }
    }

  void disconnect() {
    try {
      dbConn.close();
      }
    catch (Exception e) {
      e.printStackTrace();
      }
    }
}

If we use BasicDataSource to getConnection instead of DriverManager,
use BasicDataSource .setLogWriter.

In addition, to log these sql strings to log4j, we will redirect these messages from printwriter to log4j. Use the following code to do this.

http://www.cenqua.com/clover/eg/jboss/report/org/jboss/logging/util/LoggerWriter.html

Show lock on table in oracle and unlock them

2008-05-21T01:35:00.000-07:00

Show lock
=========
select O.object_name,
O.object_type,
S.sid,
S.serial#,
L.oracle_username,
L.os_user_name,
L.locked_mode
from dba_objects O,
v$locked_object L,
v$session S
where O.object_id = L.object_id
and S.sid = L.session_id
=============

Unlock
ALTER SYSTEM KILL SESSION 'sid, serialid';

Angry with entrecard

2008-05-05T23:12:00.000-07:00

I am very angry with entrecard system. They considers my site was spam blog.
They said that because entrecard is their system and they want to do what they want.
It is not acceptable. I will make my own system which can be better than entrecard. If we have more competition, i think they will know where they are in internet.
They delete my site but all their ads is still in my site. What do you think about them? They are all bullshit.

How to use org.apache.commons.httpclient and excute method via proxy

2008-04-12T07:07:00.000-07:00

I try to find out how to use httpclient with proxy and do not want to change java system property. But it seems that we can not find much document about it.
Finally, i have find out the solution:

First: Create httpclient
HttpClient client = new HttpClient();

set proxy
client.getHostConfiguration().setProxy(proxyhost, proxyPort);

//create authenticate
AuthScope auth = new AuthScope(proxyhost,proxyPort, AuthScope.ANY_REALM);
Credentials cred = new UsernamePasswordCredentials("","");
client.getState().setProxyCredentials(auth, cred);
=> now we can connect to internet through proxy.

If you want more information, please post comment. I will answer.

Privacy Policy

2008-03-08T18:41:00.000-08:00

Privacy Policy

The privacy of our visitors to http://thegioitinhoccuatoi.blogspot.com is important to us.

At http://thegioitinhoccuatoi.blogspot.com, we recognize that privacy of your personal information is important. Here is information on what types of personal information we receive and collect when you use visit http://thegioitinhoccuatoi.blogspot.com, and how we safeguard your information. We never sell your personal information to third parties.

Log Files

As with most other websites, we collect and use the data contained in log files. The information in the log files include your IP (internet protocal) address, your ISP (internet service provider, such as AOL or Shaw Cable), the browser you used to visit our site (such as Internet Explorer or Firefox), the time you visited our site and which pages you visited throughout our site.

Cookies and Web Beacons

We do use cookies to store information, such as your personal preferences when you visit our site. This could include only showing you a popup once in your visit, or the ability to login to some of our features, such as forums.

We also use third party advertisements on http://thegioitinhoccuatoi.blogspot.com to support our site. Some of these advertisers may use technology such as cookies and web beacons when they advertise on our site, which will also send these advertisers (such as Google through the Google AdSense program) information including your IP address, your ISP , the browser you used to visit our site, and in some cases, whether you have Flash installed. This is generally used for geotargeting purposes (showing New York real estate ads to someone in New York, for example) or showing certain ads based on specific sites visited (such as showing cooking ads to someone who frequents cooking sites).

You can chose to disable or selectively turn off our cookies or third-party cookies in your browser settings, or by managing preferences in programs such as Norton Internet Security. However, this can affect how you are able to interact with our site as well as other websites. This could include the inability to login to services or programs, such as logging into forums or accounts.
Contact Information

If users have any questions or suggestions regarding our privacy policy, please contact me through my contact form or send me an email to: testoldfriend(at) yahoo (dot) com.

Hibernate, JBoss and auto-increment field

2008-03-07T22:20:00.000-08:00

The element

This is the optional element under element. The element is used to specify the class name to be used to generate the primary key for new record while saving a new record. The element is used to pass the parameter (s) to the class. Here is the example of generator element from our first application:

In this case element do not generate the primary key and it is required to set the primary key value before calling save() method.

Here are the list of some commonly used generators in hibernate:

Generator	Description
increment	It generates identifiers of type long, short or int that are unique only when no other process is inserting data into the same table. It should not the used in the clustered environment.
identity	It supports identity columns in DB2, MySql, MS SQL Server, Sybase and HypersonicSQL. The returned identifier is of type long, short or int.
sequence	The sequence generator uses a sequence in DB2, PostgreSQL, Oracle, SAP DB, McKoi or a generator in Interbase. The returned identifier is of type long, short or int
hilo	The hilo generator uses a hi/lo algorithm to efficiently generate identifiers of type long, short or int, given a table and column (by default hibernate_unique_key and next_hi respectively) as a source of hi values. The hi/lo algorithm generates identifiers that are unique only for a particular database. Do not use this generator with connections enlisted with JTA or with a user-supplied connection.
seqhilo	The seqhilo generator uses a hi/lo algorithm to efficiently generate identifiers of type long, short or int, given a named database sequence.
uuid	The uuid generator uses a 128-bit UUID algorithm to generate identifiers of type string, unique within a network (the IP address is used). The UUID is encoded as a string of hexadecimal digits of length 32.
guid	It uses a database-generated GUID string on MS SQL Server and MySQL.
native	It picks identity, sequence or hilo depending upon the capabilities of the underlying database.
assigned	lets the application to assign an identifier to the object before save() is called. This is the default strategy if no element is specified.
select	retrieves a primary key assigned by a database trigger by selecting the row by some unique key and retrieving the primary key value.
foreign	uses the identifier of another associated object. Usually used in conjunction with a primary key association.

Therefore,
If your application is EJB, WS which create multiple threads, you should not use class=increment. Instead, you should use class=sequence or native. Or you will
encounter the problem: unique constrain violation.

Java: how to find the absolute path with white space to the resource ?

2008-02-15T23:27:00.000-08:00

For example: our program contains the resource A and class MyClass


    src
       main
          java
             MyClass
       resource
          A

When we package our program, A will be at root.
To get the absolute path to resource(for example when we want to do unit test
and load those resource for test), we will call
String path = MyClass.getClass.GetResource("/A").toURI().getPath();
And now we have the absolute path to resource.

JMock and how to mock static function and final class?

2008-02-15T22:46:00.000-08:00

Unit test is one of the "must" task when we develop software. One of the most technology we usually used is mock. As i already said in one of my previous post,
mock is the powerful technology but they have some limitation. This limitation is depend on which type of mock object( JMock, AMock, NMock ...) or the environment.
For example, if you want to test some DAO class which directly access to DBMS, you can not create mock for all JDBC class to do unit test. And in this case, we also can not do unit test because the test depend on the environment(we have to set up an DBMS just for do unit test). Therefore, we have to chose the most suitable way to test: System test, module test, review code ...
JMock like others mock object, has it own limitation. It can not mock final class, static method. It is because of its implementation. For final class, JMock can not inherit the class to create new mock class ==> can not mock this class.
With static method, because static method can not be overriden, so JMock has no way to create mock object.

So in this case, we have to think about design and test first before code
1. Case: our method use final class.

We will use pattern template method to separate code which contain instances of final class.


class OurClass {
    function myFunc() {
        my code
        ......
        URL url = new URL(href); //Can not mock URL because of final class
        url.openstream(...)
        ....
        my code


    } 
}
Our mission is test this function.

In this case, we create new class calls MockHelperTest and a factory MockHelperFactory



IHelperFactory = interface{
    IHelperTest getMockHelper;
}

MockHelperTest = class implements IHelperTest{
        
    void funcHelper() {
      ///Prepare data , result and code for mock
    }
}

and we change our class to:


class OurClass {
    public IHelperFactory factory = null; //factory can be set by unit test
    
    //If we test this function, we will 
    //pass mock factory to this class and 
    //function will call mock function 
    //instead of real code.
    //If we want to to do real code, 
    //just pass null factory, and it will
    //create real factory and then myFunc 
    //will call funcMockHelper which call 
    //real code:         
    //URL url = new URL(href); //Can not mock URL because of final class
    //url.openstream(...)

    OurClass(IHelperFactory  mockFactory) {
         if mockFactory == null ) {
              factory = new RealFactory();
         } else {
              factory = mockFactory;
         }
    }

    function myFunc() {
        
        my code
        ......
        IHelperTest helper = factory.funcHelper().getMockHelper();        
        helper.funcMockHelper();
        ....
        my code
    } 
}

And now, we can easily replace the code which can not mock which the mock code.
The same pattern for static function.

Conclusion: Think about test + design first and then code.

NUnit === How to and FAQ

2008-02-03T09:31:00.000-08:00

NUnit goal:

+ NUnit is a unit-testing framework for all .Net languages. Initially ported from JUnit, the current production release, version 2.4, is the fifth major release of this xUnit based unit testing tool for Microsoft .NET. It is written entirely in C# and has been completely redesigned to take advantage of many .NET language features, for example custom attributes and other reflection related capabilities. NUnit brings xUnit to all .NET languages
Homepage: http://www.nunit.org/index.php

+ Why do we use NUnit?
Because it makes us do unit test very easy and very efficiently

+ How to use NUnit:

1.We have to import library:
using System;
using NUnit.Framework;
2. Declare[TestFixture] attribute to class
3. Declare [SetUp] attribute for setup function. This function will be called before any call to tested function
4. Run NUnit test GUI to test assemblies

+ Some functions need for test:
Assert.AreEqual: raise exception if two object are not the same
Assert.IsTrue: raise exception if condition is false
Assert.IsFalse: raise exception if condition is true
Assert.Fail: Raise exception to notify test case fail.
....

+ How do I implement a test case for a thrown exception?
Catch the exception and if it isn't thrown call the Fail method. Fail signals the failure of a test case. Here is an example:


    public void TestArgumentOutOfRangeException() {
        ArrayList l = new ArrayList(10)
        try {
             Object o= l[l.Count];
             Fail("Should raise an ArgumentOutOfRangeException");
        } catch (ArgumentOutOfRangeException) {
        }
    }

or use the ExceptionTestCase as follows.
1) make your TestCase class a subclass of ExceptionTestCase.
2) write the test ignoring exceptions


    public void TestArgumentOutOfRangeException() {
        ArrayList l= new ArrayList(10);
        l[l.Count];
    }

3) create the TestCase:


    Test t= new ExceptionTestCase("TestArgumentOutOfRangeException", typeof(ArgumentOutOfRangeException))

Looking at this again, the first way is simpler. Sigh...

Another way:
Declare:


[Test]
[[ExpectedException( typeof( AException ) )]
void testFunction() {
...............
};

+ How do I organize my Test Cases?
Here is one way:

1. create a test namespace for each of your application namespaces. For example, for a namespace MyApp.Util define MyApp.UtilTest. Put all the fixtures for the Util namespace into this namespace.
2. in MyApp.UtilTest define a class which creates a suite with all the tests in this namespace. To do so define a class AllTests which includes a single static Suite property. Here is an example:


       public static ITest Suite {
         get {
           TestSuite suite= new TestSuite();
           suite.AddTest(Fixture1.Suite);
           suite.AddTest(Fixture2.Suite);
           return suite;
         }
       }

3. define similar AllTests classes that create higher level suites containing the suites from other test packages.

When the fixtures are in a separate test assembly the test cases don't have access to the methods and fields with internal visibility. A variation of the above convention is to put all fixtures into the application assembly itself. This gives the fixtures access to all the assembly visible methods and fields. To separate the fixture classes from the production classes put them into a separate directory that you then add to the project. This makes it easy to ship the production classes independent of the fixtures.

How do I run setup code once for all my TestCases?
Wrap the top level suite in a subclass of TestSetup. Here is a sample AllTests.Suite method:


    protected class WrappedTestSetup: TestSetup {
      public WrappedTestSetup(ITest test) : base(test) {} 
      protected override void SetUp() {
        OneTimeSetUp();
       }
      };

    public static ITest Suite {
      get {
        TestSuite suite= new TestSuite();
        ...add your tests and suites here...
        TestSetup wrapper= new WrappedTestSetup(suite);
        return wrapper;
      }
    }

+ I want to debug when a test fails
Start the test runner under the debugger and configure the debugger so that it catches the NUnit.Framework.AssertionFailedError. How you do this depends on the used IDE. Most debuggers support to stop the program when a specific exception is fired. Notice, that this will only break into the debugger when an "anticipated" assertion failed error occurs.

Steps:
1. Start C# IDE ( Visual .net ide)
2. Build project + test in debug mode.
3. Configure NUnit GUI to support VS.NET project
3. Add debug assembly to NUnit GUI.
4. Put breakpoint to where you want to debug
Finished.

Sample:


namespace NUnit.Samples.Money 
{
 using System;
 using NUnit.Framework;
 /// 
 /// 
 /// 

 /// 
 [TestFixture]
 public class MoneyTest 
 {
  private Money f12CHF;
  private Money f14CHF;
  private Money f7USD;
  private Money f21USD;
        
  private MoneyBag fMB1;
  private MoneyBag fMB2;

  /// 
  /// 
  /// 

  /// 
  [SetUp]
  protected void SetUp() 
  {
   f12CHF= new Money(12, "CHF");
   f14CHF= new Money(14, "CHF");
   f7USD= new Money( 7, "USD");
   f21USD= new Money(21, "USD");

   fMB1= new MoneyBag(f12CHF, f7USD);
   fMB2= new MoneyBag(f14CHF, f21USD);
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void BagMultiply() 
  {
   // {[12 CHF][7 USD]} *2 == {[24 CHF][14 USD]}
   Money[] bag = { new Money(24, "CHF"), new Money(14, "USD") };
   MoneyBag expected= new MoneyBag(bag);
   Assert.AreEqual(expected, fMB1.Multiply(2));
   Assert.AreEqual(fMB1, fMB1.Multiply(1));
   Assert.IsTrue(fMB1.Multiply(0).IsZero);
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void BagNegate() 
  {
   // {[12 CHF][7 USD]} negate == {[-12 CHF][-7 USD]}
   Money[] bag= { new Money(-12, "CHF"), new Money(-7, "USD") };
   MoneyBag expected= new MoneyBag(bag);
   Assert.AreEqual(expected, fMB1.Negate());
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void BagSimpleAdd() 
  {
   // {[12 CHF][7 USD]} + [14 CHF] == {[26 CHF][7 USD]}
   Money[] bag= { new Money(26, "CHF"), new Money(7, "USD") };
   MoneyBag expected= new MoneyBag(bag);
   Assert.AreEqual(expected, fMB1.Add(f14CHF));
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void BagSubtract() 
  {
   // {[12 CHF][7 USD]} - {[14 CHF][21 USD] == {[-2 CHF][-14 USD]}
   Money[] bag= { new Money(-2, "CHF"), new Money(-14, "USD") };
   MoneyBag expected= new MoneyBag(bag);
   Assert.AreEqual(expected, fMB1.Subtract(fMB2));
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void BagSumAdd() 
  {
   // {[12 CHF][7 USD]} + {[14 CHF][21 USD]} == {[26 CHF][28 USD]}
   Money[] bag= { new Money(26, "CHF"), new Money(28, "USD") };
   MoneyBag expected= new MoneyBag(bag);
   Assert.AreEqual(expected, fMB1.Add(fMB2));
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void IsZero() 
  {
   Assert.IsTrue(fMB1.Subtract(fMB1).IsZero);

   Money[] bag = { new Money(0, "CHF"), new Money(0, "USD") };
   Assert.IsTrue(new MoneyBag(bag).IsZero);
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void MixedSimpleAdd() 
  {
   // [12 CHF] + [7 USD] == {[12 CHF][7 USD]}
   Money[] bag= { f12CHF, f7USD };
   MoneyBag expected= new MoneyBag(bag);
   Assert.AreEqual(expected, f12CHF.Add(f7USD));
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void MoneyBagEquals() 
  {
   //NOTE: Normally we use Assert.AreEqual to test whether two
   // objects are equal. But here we are testing the MoneyBag.Equals()
   // method itself, so using AreEqual would not serve the purpose.
   Assert.IsFalse(fMB1.Equals(null)); 

   Assert.IsTrue(fMB1.Equals( fMB1 ));
   MoneyBag equal= new MoneyBag(new Money(12, "CHF"), new Money(7, "USD"));
   Assert.IsTrue(fMB1.Equals(equal));
   Assert.IsTrue(!fMB1.Equals(f12CHF));
   Assert.IsTrue(!f12CHF.Equals(fMB1));
   Assert.IsTrue(!fMB1.Equals(fMB2));
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void MoneyBagHash() 
  {
   MoneyBag equal= new MoneyBag(new Money(12, "CHF"), new Money(7, "USD"));
   Assert.AreEqual(fMB1.GetHashCode(), equal.GetHashCode());
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void MoneyEquals() 
  {
   //NOTE: Normally we use Assert.AreEqual to test whether two
   // objects are equal. But here we are testing the MoneyBag.Equals()
   // method itself, so using AreEqual would not serve the purpose.
   Assert.IsFalse(f12CHF.Equals(null)); 
   Money equalMoney= new Money(12, "CHF");
   Assert.IsTrue(f12CHF.Equals( f12CHF ));
   Assert.IsTrue(f12CHF.Equals( equalMoney ));
   Assert.IsFalse(f12CHF.Equals(f14CHF));
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void MoneyHash() 
  {
   Assert.IsFalse(f12CHF.Equals(null)); 
   Money equal= new Money(12, "CHF");
   Assert.AreEqual(f12CHF.GetHashCode(), equal.GetHashCode());
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void Normalize() 
  {
   Money[] bag= { new Money(26, "CHF"), new Money(28, "CHF"), new Money(6, "CHF") };
   MoneyBag moneyBag= new MoneyBag(bag);
   Money[] expected = { new Money(60, "CHF") };
   // note: expected is still a MoneyBag
   MoneyBag expectedBag= new MoneyBag(expected);
   Assert.AreEqual(expectedBag, moneyBag);
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void Normalize2() 
  {
   // {[12 CHF][7 USD]} - [12 CHF] == [7 USD]
   Money expected= new Money(7, "USD");
   Assert.AreEqual(expected, fMB1.Subtract(f12CHF));
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void Normalize3() 
  {
   // {[12 CHF][7 USD]} - {[12 CHF][3 USD]} == [4 USD]
   Money[] s1 = { new Money(12, "CHF"), new Money(3, "USD") };
   MoneyBag ms1= new MoneyBag(s1);
   Money expected= new Money(4, "USD");
   Assert.AreEqual(expected, fMB1.Subtract(ms1));
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void Normalize4() 
  {
   // [12 CHF] - {[12 CHF][3 USD]} == [-3 USD]
   Money[] s1 = { new Money(12, "CHF"), new Money(3, "USD") };
   MoneyBag ms1= new MoneyBag(s1);
   Money expected= new Money(-3, "USD");
   Assert.AreEqual(expected, f12CHF.Subtract(ms1));
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void Print() 
  {
   Assert.AreEqual("[12 CHF]", f12CHF.ToString());
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void SimpleAdd() 
  {
   // [12 CHF] + [14 CHF] == [26 CHF]
   Money expected= new Money(26, "CHF");
   Assert.AreEqual(expected, f12CHF.Add(f14CHF));
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void SimpleBagAdd() 
  {
   // [14 CHF] + {[12 CHF][7 USD]} == {[26 CHF][7 USD]}
   Money[] bag= { new Money(26, "CHF"), new Money(7, "USD") };
   MoneyBag expected= new MoneyBag(bag);
   Assert.AreEqual(expected, f14CHF.Add(fMB1));
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void SimpleMultiply() 
  {
   // [14 CHF] *2 == [28 CHF]
   Money expected= new Money(28, "CHF");
   Assert.AreEqual(expected, f14CHF.Multiply(2));
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void SimpleNegate() 
  {
   // [14 CHF] negate == [-14 CHF]
   Money expected= new Money(-14, "CHF");
   Assert.AreEqual(expected, f14CHF.Negate());
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void SimpleSubtract() 
  {
   // [14 CHF] - [12 CHF] == [2 CHF]
   Money expected= new Money(2, "CHF");
   Assert.AreEqual(expected, f14CHF.Subtract(f12CHF));
  }
 }
}

How to use interface correctly?

2008-01-21T21:59:00.000-08:00

A: IparentInterface;
B: IchildInterface;

This code will generate error:

B: = IchildInterface (A); ==> will generate expected error whenever we access to B. At the time we down-cast A, we do not have exception.

The correct code should be:

A.QueryInterface (IchildInterface, B);

So the lesson here: Always use QueryInterface when you want to access other inteface of the same object. Do not cast Interface except you really know what you do.

Delphi Memory Leak

2008-01-21T21:47:00.000-08:00

I have a lession learn today when I work with delphi. We should avoid using Tlist to store object which implemented any interfaces.

We should use TinterfaceList instead of Tlist. The reason is: Tlist use pointer to reference to object. When we get an object and from that object we get one of its interfaces. This interface is not referenced by anyone (although Tlist use pointer to point to the object) .Therefor when go out of variable scope, this interface is freed automatically.

Hereunder is the sample explaining the lession.


InterfaceA = Iinterface 
            Procedure methodA;
End;


ClassA = class(TinterfaceObject, InterfaceA)
Procedure methodA;
End;


Var 
A: TOBbject
List: Tlist;
Begin
List := Tlist.Create;
List.Add(A);
methodX;
methodY; ==> Exception occur here.
End.

Procedure methodX;
Var
   X: InterfaceA;
   Obj: ClassA;
Begin
   Obj := Tlist.Items[0];
  Obj.QueryInterface(InterfaceA, X); ==> Get interface here. Reference count increased to 1
End; ==> Interface will release reference count. Reference count = 0 ==> Obj is freed.
 
Procedure methodY;
Begin
Obj := Tlist.Items[0];
            Obj.methodA; ç Exception because of object is freed.
End;


Hope this lession lean can help you avoid unexpected exception.

Does java have memory leak?

2008-01-21T21:27:00.000-08:00

According to the IBM's artical, JAVA also has problems of memory leak event though they have its own memory garbage collector. The main reason is that: design is not good.

I. How memory garbage collector work?

The job of the garbage collector is to find objects that are no longer needed by an application and to remove them when they can no longer be accessed or referenced. The garbage collector starts at the root nodes, classes that persist throughout the life of a Java application, and sweeps though all of the nodes that are referenced. As it traverses the nodes, it keeps track of which objects are actively being referenced. Any classes that are no longer being referenced are then eligible to be garbage collected. The memory resources used by these objects can be returned to the Java virtual machine (JVM) when the objects are deleted.

II. How it leaks?

"An object is only counted as being unused when it is no longer referenced" ==> The garbage collector can not do its job if an object is referenced by others object for a very long time even though it is no longer needed. It can lead to a terrible problems: out of memory.

Example:

Object A = new Object; //A exists until program terminate.
During program life cycle, an object B is created and referenced by A
Because programmer believe in garbage collector, so they do not free memory manually,
but let garbage collector do it for him ==> B exist until A freed.
If program create many and many B for its life ==> all B will exist too ==> cause out of memory.

It is the obvious example of problem of memory leak. So we do not believe in Garbage Collector 100%. Let free object manually by assigned unused object to nil.

III.Preventing memory leaks

You can prevent memory leaks by watching for some common problems. Collection classes, such as hashtables and vectors, are common places to find the cause of a memory leak. This is particularly true if the class has been declared static and exists for the life of the application.

Another common problem occurs when you register a class as an event listener without bothering to unregister when the class is no longer needed. Also, many times member variables of a class that point to other classes simply need to be set to null at the appropriate time.

Hope i can help you in avoid memory leak in java.

How to get the path and filename of executable file of current process (or running process)?

2008-01-19T04:57:00.000-08:00

How to get the path + file name of the current process or the running process?
There are two way to this:

I will illustrate idea in delphi

1. You can call the API function: GetProcessImageFileName

Declare prototype:

interface
function GetProcessImageFileNameA(hProcess: THandle; lpImageFileName: LPSTR;
nSize: DWORD): DWORD; stdcall;
{$EXTERNALSYM GetProcessImageFileNameA}
function GetProcessImageFileNameW(hProcess: THANDLE; lpImageFileName: LPWSTR;
nSize: DWORD): DWORD; stdcall;
{$EXTERNALSYM GetProcessImageFileNameW}
function GetProcessImageFileName(hProcess: THANDLE; lpImageFileName: LPTSTR;
nSize: DWORD): DWORD; stdcall;
{$EXTERNALSYM GetProcessImageFileName}

implementation

function GetProcessImageFileNameA(hProcess: THandle; lpImageFileName: LPSTR;
nSize: DWORD): DWORD; stdcall; external 'Psapi.dll' name 'GetProcessImageFileNameA';
function GetProcessImageFileNameW(hProcess: THANDLE; lpImageFileName: LPWSTR;
nSize: DWORD): DWORD; stdcall; external 'Psapi.dll' name 'GetProcessImageFileNameW';
function GetProcessImageFileName(hProcess: THANDLE; lpImageFileName: LPTSTR;
nSize: DWORD): DWORD; stdcall; external 'Psapi.dll' name 'GetProcessImageFileNameA';

Calling function:

+ GetWindowThreadProcessId(WindowHandle, @processID); ==> get processid
+ processHandle := OpenProcess(PROCESS_TERMINATE or PROCESS_QUERY_INFORMATION,
False, processID); ==> Get process handler
+ GetProcessImageFileName(processHandle , bufferout, lenghtofbufferout)

2. The second way:

Call function GetModuleFileName with first parameter nil
+ GetModuleFileName(0, Buffer, 1024); <== Buffer contain the current path of current process.

Delphi --- problem when using function RegisterClass

2007-12-30T04:39:00.000-08:00

Delphi has a feature: serialize/deserialize object. It means you can save/load object at runtime.
To do this, your classes have to inherit from TPersistent and you have to register this class to system. To register a class you can you function: RegisterClass. After registering successfully, you can read or write object to the stream. But there are problems: you can not know if class is registered successfully till you call function: getclass or findclass, and you can not know why it is not succeeded. It takes me 1 day to find why it can not register. All i have to do is debug delphi system to find out the reason. And i want to share my experience when i face this problem. The reason is: the class belongs to a group. But this group is deactivated when delphi create group. So this class can not be registered. All we have to do in this case is activate group again by calling function activategroup(class belong to group). Hope i can help you to save time when solve this problem

HTML Parser -- A good html parser.

2007-12-30T04:25:00.000-08:00

I have found many parser used in my projects. By asking google, we can find some parsers. But after try many parsers, i can tell you that HTMLPaser is the one should be considered.
Website: htmlparser.sourceforge.net
HTMLParser is a parser which can help you parser html file. It has 2 features which i like very much
1. Extracter: It can help us extract information from html file very easily with many filters.
All we have to do is call parser.parse(Filter). A filter can be an individual filter or it can be predicate which combine many filter(and , or , not, user-defined filter) to help you extract as much as possible information.
2. Visitor pattern: With visitor pattern you can do what you want with a tag when parser traverse to that tag. You can change the attribute of node, get information, ... whatever you want.
Besides, htmlparser is well-structured, so it can speed-up coding, maintain easily.

If you want to find a good html parser, why don't give it a try.
PS: You should try html parser version 2.0

Introduction to blogvertise

2007-12-28T07:37:00.000-08:00

Today, i surf net and found an interesting website. You know i love to write blog, and share my experiences i met in my work. It is my hobby. But if i can earn money from those posts,it is really interesting. And Blogvertise is what i want to tell you. You can earn money from your blogs with your posts. The re quirement is simple. You get your email, follow their instruction to write a blog with some criteria. And wait for your money.

Hope it will help you about blogvertise.

Website: Blogvertise

Java: How to load class dynamically (on fly)?

2007-12-09T06:27:00.000-08:00

I face a problem of loading class dynamically in JAVA. How to do this?
After do research in internet with many site, i finally find out the way to do this.
In order to save your time on researching problem, i will show you how to do this.

To understand the framework, we have to know following knowledge:
1. Class loader and how does it work

So how does class loader work:

As shown in Figure 1, the bootstrap class loader (BS) -- Native implemented into JVM -- loads the classes from the JVM, as well as extensions to the JDK. The system class loader (CP) loads all of the classes provided by the CLASSPATH environment variable or passed using the -classpath argument to the java command. Finally we have several additional class loaders, where A1-3 are children of the CP, and B1-2 are children of A3. Every class loader (except BS) has a parent class loader, even if no parent is provided explicitly; in the latter case, the CP is automatically set as the parent.

Rule for class loader:

1. lass loaders are hierarchically organized, where each one has a parent class loader, except the bootstrap class loader (the root).

2. Class loaders should (practically: must) delegate the loading of a class to the parent, but a custom class loader can define for itself when it should do so.

3. A class is defined by its class type and the effective class loader.

4. A class is only loaded once and then cached in the class loader to ensure that the byte code cannot change.

5. Any symbolic links are loaded by the effective class loader (or one of its ancestors), if this is not already done. The JVM can defer this resolution until the class is actually used.

6. An upcast of an instance to another class fails when the class of the instance and the class of the symbolic link do not match (meaning their class loaders do not match).

But another problem is: Classes are loaded by many class loader. How can it distinguish them? JAVA do it by combine qualified name: (Class name, Package name, Class loader) --> With the same classes, but loaded by different class loader, it can not be used together. We can check it easily by using URLClassLoader and sun.misc.Launcher$AppClassLoader(class loader which load main function).

When i work with JAX-WS, i see that we can dynamically add custom handler to WS, and JAX-WS will call our handler. It make me think a lot. How can it do this?
The answer is: JAXWS has its own class loader. And the class loader will find and call our handler.

Now we will make a framework which can allow us load custom handler like JAX-WS at run time.

1. We define a interface for handler. This handler is packaged in jar file.
2. User will define class which implemented the interface handler. The implement class is packaged in other jar file.
3. Our framework will scan those jar file and load all custom handlers, and call it sequentially.

The most difficult part is load all custom handlers and call it at run time.
To solve the problem, we will make our own class loader:


public class CustomLoader extends URLClassLoader
{ 
    static public ClassLoader MainClassLoader = null;
 
    protected CustomLoader (URL[] urls, ClassLoader parent) {
        super(urls, parent);        
    }
    
    public Class loadClass(String name) throws ClassNotFoundException {
        //Get system class loader        
        try { 
         if (MainClassLoader != null) {
          return MainClassLoader.loadClass(name);
         } else {
          return super.loadClass(name);
         }
        } catch (ClassNotFoundException e) {
         return super.loadClass(name);
        }        
    }

    protected Class findClass(String name) throws ClassNotFoundException {
        Class clas = super.findClass(name);
        return clas;
    }
}

The variable MainClassLoader point to sun.misc.Launcher$AppClassLoader( the class loader which call our main function)

How to used this custom class loader?


void main() {
   ClassLoader base = ClassLoader.getSystemClassLoader(); //get the boot trap class loader
   CustomLoader loader = new CustomLoader(urls, base.getParent()); //create our class loader
   loader.MainClassLoader = Thread.currentThread().getContextClassLoader(); //set the  //application class loader to our class loader. We have to do this because when java //compile and link our code, it used class loader: sun.misc.Launcher$AppClassLoader. 

Class handlerClass = loader.loadClass("CustomHandler");
Object handler =  handlerClass.newInstance(); (1)
Handler customHandler = (Handler)handler; (2)
}

(1) --> implement class is loaded by our custom class loader.
(2) --> Handler is loaded by sun.misc.Launcher$AppClassLoader

Thank to this code:

public Class loadClass(String name) throws ClassNotFoundException {
//Get system class loader
try {
if (MainClassLoader != null) {
return MainClassLoader.loadClass(name);
} else {
return super.loadClass(name);
}
} catch (ClassNotFoundException e) {
return super.loadClass(name);
}
}

We delegate loading class to MainClassLoader first: because in linkage phase, MainClassLoader load all symbols in our source code. If we do not do this, we will violate the rule 6. If MainClassLoader does not load them, we will delegate to parent or we violate rule 2. If both do not load class, we can do it by ourselves. But in this case we do not implement this(for simple).

So with our custom class loader, (2) can be used without ClassCastException.

So i hope , you can make a framework easily with the custom class loader as i explain.

Memory Management -- Part 6

2007-11-22T11:32:00.000-08:00

Memory Management

Note the page attributes from the output of the first instance. The functions are marked read-only, as expected. The unshared variable is also marked read-only. This is because Windows NT tries to share the data space also. As described earlier, such pages are marked for copy-on-write, and as soon as the process modifies any location in the page, the process gets a private copy of the page to write to. The other page attributes show that the PTE is valid, the page is a user-mode page, and nobody has modified the page so far.

Now, compare the output from the first instance with the output from the second instance when it loaded the MYDLL.DLL at a base address different from that in the first instance. As expected, the virtual addresses of all the memory sections are different than those for the first instance. The physical addresses are the same except for the physical address of the relocatable function. This demonstrates that the code pages are marked as copy-on-write, and when the loader modifies the code pages while performing relocation, the process gets a private writable copy. Our nonrelocatable function does not need any relocation; hence, the corresponding pages are not modified. The second instance can share these pages with the first instance and hence has the same physical page address.

To cancel out the effects of relocation, the second instance loads MYDLL.DLL at the same base address as that in the first instance. Yup! Now, the virtual address matches the ones from the first instance. Note that the physical address for the relocatable function also matches that in the output from the first instance. The loader need not relocate the function because the DLL is loaded at the preferred base address. This allows more memory sharing and provides optimal performance. It’s reason enough to allocate proper, nonclashing preferred base addresses for your DLLs.

This ideal share-all situation ceases to exist as soon as a process modifies some memory location. Other processes cannot be allowed to view these modifications. Hence, the modifying process gets its own copy of the page The second instance of the sample program demonstrates this by modifying the data variables and a byte at the start of the nonrelocatable function. The output shows that the physical address of the nonrelocatable doesn’t match with the first instance. The nonrelocatable function is not modified by the loader, but it had the same effect on sharing when we modified the function. The shared variable remains a shared variable. Its physical address matches that in the first instance because all the processes accessing a shared variable are allowed to see the modifications made by other processes. But the nonshared variable has a different physical address now. The second instance cannot share the variable with the first instance and gets its own copy. The copy was created by the system page fault handler when we tried to write to a read-only page and the page was also marked for copy-on-write. Note that the page is now marked read-write. Hence, further writes go through without the operating system getting any page faults. Also, note that the modified pages are marked as dirty by the processor.

SWITCHING CONTEXT

As we saw earlier, Windows NT can switch the memory context to another process by setting the appropriate page table directory. The 80386 processor requires that the pointer to the current page table directory be maintained in the CR3 register. Therefore, when the Windows NT scheduler wants to perform a context switch to another process, it simply sets the CR3 register to the page table directory of the concerned process.

Windows NT needs to change only the memory context for some API calls such as VirtualAllocEx(). The VirtualAllocEx() API call allocates memory in the memory space of a process other than the calling process. Other system calls that require memory context switch are ReadProcessMemory() and WriteProcessMemory(). The ReadProcessMemory() and WriteProcessMemory() system calls read and write, respectively, memory blocks from and to a process other than the calling process. These functions are used by debuggers to access the memory of the process being debugged. The subsystem server processes also use these functions to access the client process’s memory. The undocumented KeAttchProcess() function from the NTOSKRNL module switches the memory context to specified process. The undocumented KeDetachProcess() function switches it back. In addition to switching memory context, it also serves as a notion of current process. For example, if you attach to a particular process and create a mutex, it will be created in the context of that process. The prototypes for KeAttachProcess() and KeDetachProcess() are as follows:

NTSTATUS KeAttachProcess(PEB *);

NTSTATUS KeDetachProcess ();

Another place where a call to the KeAttachProcess() function appears is the NtCreateProcess() system call. This system call is executed in the context of the parent process. As a part of this system call, Windows NT needs to map the system DLL (NTDLL.DLL) in the child process’s address space. Windows NT achieves this by calling KeAttachProcess() to switch the memory context to the child process. After mapping the DLL, Windows NT switches back to the parent process’s memory context by calling the KeDetachProcess() function.

The following sample demonstrates how you can use the KeAttachProcess() and KeDetachProcess() functions. The sample prints the page directories for all the processes running in the system. The complete source code is not included. Only the relevant portion of the code is given. Because these functions can be called only from a device driver, we have written a device driver and provided an IOCTL that demonstrates the use of this function. We are giving the function that is called in response to DeviceIoControl from the application. Also, the output of the program is shown in kernel mode debugger’s window (such as SoftICE). Getting the information back to the application is left as an exercise for the reader.

void DisplayPageDirectory(void *Peb)

{

unsigned int *PageDirectory =

(unsigned int *)0xC0300000;

int i;

int ctr=0;

KeAttachProcess(Peb);

for (i = 0; i < 1024; i++) {

if (PageDirectory[i]&0x01) {

if ((ctr%8) == 0)

DbgPrint(" \n");

DbgPrint("%08x ", PageDirectory[i]&0xFFFFF000);

ctr++;

}

}

DbgPrint("\n\n");

KeDetachProcess();

}

The DisplayPageDirectory() function accepts the PEB for the process whose page directory is to be printed. The function first calls the KeAttachProcess() function with the given PEB as the parameter. This switches the page directory to the desired one. Still, the function can access the local variables because the kernel address space is shared by all the processes. Now the address space is switched, and the 0xC030000 address points to the page directory to be printed. The function prints the 1024 entries from the page directory and then switches back to the original address space using the KeDetachProcess() function.

void DisplayPageDirectoryForAllProcesses()

{

PLIST_ENTRY ProcessListHead, ProcessListPtr;

ULONG BuildNumber;

ULONG ListEntryOffset;

ULONG NameOffset;

BuildNumber=NtBuildNumber & 0x0000FFFF;

if ((BuildNumber==0x421) || (BuildNumber==0x565)) { // NT

3.51 or NT 4.0

ListEntryOffset=0x98;

NameOffset=0x1DC;

} else if (BuildNumber==0x755) {// Windows 2000 beta2

ListEntryOffset=0xA0;

NameOffset=0x1FC;

} else {

DbgPrint("Unsupported NT Version\n");

return;

}

ProcessListHead=ProcessListPtr=(PLIST_ENTRY)(((char

*)PsInitialSystemProcess)+ListEntryOffset);

while (ProcessListPtr->Flink!=ProcessListHead) {

void *Peb;

char ProcessName[16];

Peb=(void *)(((char *)ProcessListPtr)-

ListEntryOffset);

memset(ProcessName, 0, sizeof(ProcessName));

memcpy(ProcessName, ((char *)Peb)+NameOffset, 16);

DbgPrint("**%s Peb @%x** ", ProcessName, Peb);

DisplayPageDirectory(Peb);

ProcessListPtr=ProcessListPtr->Flink;

}

}

The DisplayPageDirectoryForAllProcesses() function calls the DisplayPageDirectory() function for each process in the system. All the processes running in a system are linked in a list. The function gets hold of the list of the processes from the PEB of the initial system process. The PsInitialSystemProcess variable in NTOSKRNL holds the PEB for the initial system process. The process list node is located at an offset of 0x98 (0xA0 for Windows NT 5.0) inside the PEB. The process list is a circular linked list. Once you get hold of any node in the list, you can traverse the entire list. The DisplayPageDirectoryForAllProcesses() function completes a traversal through the processes list by following the Flink member, printing the page directory for the next PEB in the list every time until it reaches back to the PEB it started with. For every process, the function first prints the process name that is stored at a version-dependent offset within the PEB and then calls the DisplayPageDirectory() function to print the page directory.

Here, we list partial output from the sample program. Please note a couple of things in the following output. First, every page directory has 50-odd valid entries while the page directory size is 1024. The remaining entries are invalid, meaning that the corresponding page tables are either not used or are swapped out. In other words, the main memory overhead of storing page tables is negligible because the page tables themselves can be swapped out. Also, note that the page directories have the same entries in the later portion of the page directory. This is because this part represents the kernel portion shared across all processes by using the same set of page tables for the kernel address range.

Listing 4-12: Displaying page directories: output

**System Peb @fdf06b60**

00500000 008cf000 008ce000 00032000 00034000 00035000 ... ... ...

00040000 00041000 00042000 00043000 00044000 00045000 ... ... ...

00048000 00049000 0004a000 0004b000 0004c000 0004d000 ... ... ...

00050000 00051000 00052000 00053000 00054000 00055000 ... ... ...

00058000 00059000 0005a000 0005b000 0005c000 0005d000 ... ... ...

00020000 00021000 00023000 0040b000 0040c000 0040d000 ... ... ...

00410000 00411000 00412000 00413000 00414000 00415000 ... ... ...

**smss.exe Peb @fe2862e0**

00032000 00034000 00035000 00033000 00e90000 00691000 ... ... ...

00043000 00044000 00045000 00046000 00047000 00048000 ... ... ...

0004b000 0004c000 0004d000 0004e000 0004f000 00050000 ... ... ...

00053000 00054000 00055000 00056000 00057000 00058000 ... ... ...

0005b000 0005c000 0005d000 0005e000 0005f000 00020000 ... ... ...

0040b000 0040c000 0040d000 0040e000 0040f000 00410000 ... ... ...

00413000 00414000 00415000 00416000 00031000

... ... ...

**winlogon.exe Peb @fe27dde0**

00032000 00034000 00035000 00033000 00be1000 00953000 ... ... ...

00043000 00044000 00045000 00046000 00047000 00048000 ... ... ...

0004b000 0004c000 0004d000 0004e000 0004f000 00050000 ... ... ...

00053000 00054000 00055000 00056000 00057000 00058000 ... ... ...

0005b000 0005c000 0005d000 0005e000 0005f000 00020000 ... ... ...

0040b000 0040c000 0040d000 0040e000 0040f000 00410000 ... ... ...

00413000 00414000 00415000 00416000 00031000

... ... ...

DIFFERENCES BETWEEN WINDOWS NT AND WINDOWS 95/98

Generally, the memory management features offered by Windows 95/98 are the same as those in Windows NT. Windows 95/98 also offers 32-bit flat separate address space for each process. Features such as shared memory are still available. However, there are some differences. These differences are due to the fact that Windows 95/98 is not as secure as Windows NT. Many times, Windows 95/98 trades off security for performance reasons. Windows 95/98 still has the concept of user-mode and kernel-mode code. The bottom 3GB is user-mode space, and the top 1GB is kernel-mode space. But the 3GB user-mode space can be further divided into shared space and private space for Windows 95/98. The 2GB to 3GB region is the shared address space for Windows 95/98 processes. For all processes, the page tables for this shared region point to the same set of physical pages.

All the shared DLLs are loaded in the shared region. All the system DLLs–for example, KERNEL32.DLL and USER32.DLL–are shared DLLs. Also, a DLL’s code/data segment can be declared shared while compiling the DLL, and the DLL will get loaded in the shared region. The shared memory blocks are also allocated space in the shared region. In Windows 95/98, once a process maps a shared section, the section is visible to all processes. Because this section is mapped in shared region, other processes need not map it separately.

There are advantages as well as disadvantages of having such a shared region. Windows 95/98 need not map the system DLLs separately for each process; the corresponding entries of page table directory can be simply copied for each process. Also, the system DLLs loaded in shared region can maintain global data about all the processes and separate subsystem processes are not required. Also, most system calls turn out to be simple function calls to the system DLLs, and as a result are very fast. In Windows NT, most system calls either cause a context switch to kernel mode or a context switch to the subsystem process, both of which are costly operations. For developers, loading system DLLs in a shared region means that they can now put global hooks for functions in system DLLs.

For all these advantages, Windows 95/98 pays with security features. In Windows 95/98, any process can access all the shared data even if it has not mapped it. It can also corrupt the system DLLs and affect all processes.

SUMMARY

In this chapter, we discussed the memory management of Windows NT from three different perspectives. Memory management offers programmers a 32-bit flat address space for every process. A process cannot access another process’s memory or tamper with it, but two processes can share memory if they need to. Windows NT builds its memory management on top of the memory management facilities provided by the microprocessor. The 386 (and above) family of Intel microprocessors provides support for segmentation plus paging. The address translation mechanism first calculates the virtual address from the segment descriptor and the specified offset within the segment. The virtual address is then converted to a physical address using the page tables. The operating system can restrict access to certain memory regions by using the security mechanisms that are provided both at the segment level and the page level.

Windows NT memory management provides the programmer with flat address space, data sharing, and so forth by selectively using the memory management features of the microprocessor. The virtual memory manager takes care of the paging and allows 4GB of virtual address space for each process, even when the entire system has much less physical memory at its disposal. The virtual memory manager keeps track of all the physical pages in the system through the page frame database (PFD). The system also keeps track of the virtual address space for each process using the virtual address descriptor (VAD) tree. Windows NT uses the copy-on-write mechanism for various purposes, especially for sharing the DLL data pages. The memory manager has an important part in switching the processor context when a process is scheduled for execution. Windows 95/98 memory management is similar to Windows NT memory management with the differences being due to the fact that Windows 95/98 is not as security conscious as Windows NT.

Memory Management -- Part 5

2007-11-22T11:31:00.002-08:00

Memory Management

IMPACT ON HOOKING

Now we’ll look at the impact of the memory management scheme explained in the last section in the area of hooking DLL API calls. To hook a function from a DLL, you need to change the first few bytes from the function code. As you saw earlier, the DLL code is shared by all processes and is write protected so that a misbehaving process cannot affect other processes. Does this mean that you cannot hook a function in Windows NT? The answer is, “Hooking is possible under Windows NT, but you need to do a bit more work to comply with stability requirements.” Windows NT provides a system call, VirtualProtect, that you can use to change page attributes. Hence, hooking is now a two-step process: Change the attributes of the page containing DLL code to read-write, and then change the code bytes.

Copy-on-Write
“Eureka!” you might say, “I violated Windows NT security. I wrote to a shared page used by other processes also.” No! You did not do that. You changed only your copy of the DLL code. The DLL code page was being shared while you did not write to the page. The moment you wrote on that page, a separate copy of it was made, and the writes went to this copy. All other processes are safely using the original copy of the page. This is how Windows NT protects processes from each other while consuming as few resources as possible.

The VirtualProtect() function does not mark the page as read-write–it keeps the page as read-only. Nevertheless, to distinguish this page from normal read-only pages, it is marked for copy-on-write. Windows NT uses one of the available PTE bits for doing this. When this page is written onto, because it is a read-only page, the processor raises a page fault exception. The page fault handler makes a copy of the page and modifies the page table of the faulting process accordingly. The new copy is marked as read-write so that the process can write to it.

Windows NT itself uses the copy-on-write mechanism for various purposes. The DLL data pages are shared with the copy-on-write mark. Hence, whenever a process writes to a data page, it gets a personal copy of it. Other processes keep sharing the original copy, thus maximizing the sharing and improving memory usage.

A DLL may be loaded in memory at different linear address for different processes. The memory references–for example, address for call instruction, address for a memory to register move instruction, and so on–in the DLL need to be adjusted (patched) depending on the linear address where the DLL gets loaded. This process is called as relocating the DLL. Obviously, relocation has to be done separately for each process. While relocating, Windows NT marks the DLL code pages as copy-on-write temporarily. Thus, only the pages requiring page relocation are copied per process. Other pages that do not have memory references in them are shared by all processes.

This is the reason Microsoft recommends that a DLL be given a preferred base address and be loaded at that address. The binding of the DLL to a specific base address ensures that the DLL need not be relocated if it is loaded at the specified base address. Hence, if all processes load the DLL at the preferred base address, all can share the same copy of DLL code.

The POSIX subsystem of Windows NT uses the copy-on-write mechanism to implement the fork system call. The fork system call creates a new process as a child of a calling process. The child process is a replica of the parent process, and it has the same state of code and data pages as the parent. Since these are two different processes, the data pages should not be shared by them. However, generally it is wasteful to make a copy of the parent’s data pages because in most cases the child immediately invokes the exec system call. The exec system call discards the current memory image of the process, loads a new executable module, and starts executing the new executable module. To avoid copying the data pages, the fork system call marks the data pages as copy-on-write. Hence, a data page is copied only if the parent or the child writes to it.

Copy-on-write is an extremely important concept contributing to the efficiency of NT memory management.

The following sample program demonstrates how copy-on-write works. By running two instances of the program, you can see how the concepts described in this section work. The application loads a DLL, which contains two functions and two data variables. One function does not refer to the outside world, so no relocations are required for it. The other function accesses one global variable, so it contains relocatable instructions or instructions that need relocation. One data variable is put in a shared data section so it will be shared across multiple instances of DLL. One variable is put in a default data section. The two functions are put in separate code sections just to make them page aligned.

When you run the first instance of the application, the application loads and prints the physical addresses of two functions and two data variables. After this, you run the second instance of the same application. In the second instance, the application arranges to load the DLL at a different base address than that of the first instance. Then it prints the physical addresses of two functions and two data variables. Next, the application arranges to load the DLL at the same base address as that of the first instance. In this case, all physical pages are seen to be shared. Next, the application modifies the shared and nonshared variable and modifies the first few bytes of one function, and it prints the physical addresses for two functions and two variables again. We first discuss the code for this sample program and then describe how the output from the sample program demonstrates memory sharing and the effects of the copy-on-write mechanism.

Listing 4-8: SHOWPHYS.C

#include

#include

#include "gate.h"

#include "getphys.h"

HANDLE hFileMapping;

/* Imported function/variable addresses */

static void *NonRelocatableFunction = NULL;

static void *RelocatableFunction = NULL;

static void *SharedVariable = NULL;

static void *NonSharedVariable = NULL;

HINSTANCE hDllInstance;

The initial portion of the file contains the header inclusion and global variable definitions. The program demonstrates the use of various page attributes, especially to implement the copy-on-write mechanism. As described earlier, the program uses four different types of memory sections. The pointers to the four different types of memory sections are defined as global variables. The hDllInstance stores the instance of the instance handle of the DLL that contains the different kind of memory sections used in this demonstration.

/* Loads MYDLL.DLL and initializes addresses of

* imported functions/variables from MYDLL.DLL and

* locks the imported areas

*/

int LoadDllAndInitializeVirtualAddresses()

{

hDllInstance = LoadLibrary("MYDLL.DLL");

if (hDllInstance == NULL) {

printf("Unable to load MYDLL.DLL\n");

return -1;

}

printf("MYDLL.DLL loaded at base address = %x\n",

hDllInstance);

NonRelocatableFunction =

GetProcAddress(GetModuleHandle("MYDLL"),

"_NonRelocatableFunction@0");

RelocatableFunction =

GetProcAddress(GetModuleHandle("MYDLL"),

"_RelocatableFunction@0");

SharedVariable =

GetProcAddress(GetModuleHandle("MYDLL"),

"SharedVariable");

NonSharedVariable =

GetProcAddress(GetModuleHandle("MYDLL"),

"NonSharedVariable");

if((!NonRelocatableFunction) ||

(!RelocatableFunction) ||

(!SharedVariable) ||

(!NonSharedVariable)) {

printf("Unable to get the virtual addresses for"

"imports from MYDLL.DLL\n");

FreeLibrary(hDllInstance);

HDllInstance = 0;

return -1;

}

VirtualLock(NonRelocatableFunction, 1);

VirtualLock(RelocatableFunction, 1);

VirtualLock(SharedVariable, 1);

VirtualLock(NonSharedVariable, 1);

return 0;

}

The four different types of memory sections that we use for the demonstration reside in MYDLL.DLL. The LoadDllAndInitializeVirtualAddresses() function loads MYDLL.DLL in the calling process’s address space and initializes the global variables to point to different types of memory sections in the DLL. The function uses the GetProcAddress() function to get hold of pointers to the exported functions and variables in MYDLL.DLL. The function stores the instance handle for MYDLL.DLL in a global variable so that the FreeDll() function can later use it to unload the DLL. The function also locks the different memory sections so that the pages are loaded in memory and the page table entries are valid. Generally, Windows NT does not load the page table entries unless the virtual address is actually accessed. In other words, the memory won’t be paged in unless accessed. Also, the system can page out the memory that is not used for some time, again marking the page table entries as invalid. We use the VirtualLock() function to ensure that the pages of interest are always loaded and the corresponding page table entries remain valid.

/* Unlocks the imported areas and frees the MYDLL.DLL

*/

void FreeDll()

{

VirtualUnlock(NonRelocatableFunction, 1);

VirtualUnlock(RelocatableFunction, 1);

VirtualUnlock(SharedVariable, 1);

VirtualUnlock(NonSharedVariable, 1);

FreeLibrary(hDllInstance);

HDllInstance = 0;

NonRelocatableFunction = NULL;

RelocatableFunction = NULL;

SharedVariable = NULL;

NonSharedVariable = NULL;

}

The FreeDll() function uses the VirtualUnlock() function to unlock the memory locations locked by the LoadDllAndInitializeVirtualAddresses() function. The function unloads MYDLL.DLL after unlocking the memory locations from the DLL. As the DLL is unloaded, the global pointers to the memory sections in the DLL become invalid. The function sets all these pointers to NULL according to good programming practice.

/* Converts the page attributes in readable form

*/

char *GetPageAttributesString(unsigned int PageAttr)

{

static char buffer[100];

strcpy(buffer, "");

strcat(buffer, (PageAttr&0x01)? "P ": "NP ");

strcat(buffer, (PageAttr&0x02)? "RW ": "R ");

strcat(buffer, (PageAttr&0x04)? "U ": "S ");

strcat(buffer, (PageAttr&0x40)? "D ": " ");

return buffer;

}

The GetPageAttributesString() function returns a string with characters showing the page attributes given the page attribute flags. The LSB in the page attributes indicates whether the page is present in memory or the page table entry is invalid. This information is printed as P or NP, which stands for present or not present. Similarly, R or RW means a read-only or read-write page; S or U means a supervisor-mode or a user-mode page; and D means a dirty page. The various page attributes are represented by different bits in the PageAttr parameter to this function. The function checks the bits and determines whether the page possesses the particular attributes.

/* Displays virtual to physical address mapping

*/

int DisplayVirtualAndPhysicalAddresses()

{

DWORD pNonRelocatableFunction = 0;

DWORD pRelocatableFunction = 0;

DWORD pSharedVariable = 0;

DWORD pNonSharedVariable = 0;

DWORD aNonRelocatableFunction = 0;

DWORD aRelocatableFunction = 0;

DWORD aSharedVariable = 0;

DWORD aNonSharedVariable = 0;

printf("\nVirtual to Physical address mapping\n");

printf("\n------------------------------------\n");

printf("Variable/function Virtual Physical Page\n");

printf(" Address Address Attributes\n");

printf("--------------------------------------\n");

GetPhysicalAddressAndPageAttributes(

NonRelocatableFunction,

&pNonRelocatableFunction, &aNonRelocatableFunction);

GetPhysicalAddressAndPageAttributes(

RelocatableFunction,

&pRelocatableFunction, &aRelocatableFunction);

GetPhysicalAddressAndPageAttributes(

SharedVariable,

&pSharedVariable,

&aSharedVariable);

GetPhysicalAddressAndPageAttributes(

NonSharedVariable,

&pNonSharedVariable,

&aNonSharedVariable);

printf("NonRelocatableFunction\t %8x\t %8x\t %s\n",

NonRelocatableFunction,

pNonRelocatableFunction,

GetPageAttributesString(

aNonRelocatableFunction));

printf("RelocatableFunction\t %8x\t %8x\t %s\n",

RelocatableFunction,

pRelocatableFunction,

GetPageAttributesString(

aRelocatableFunction));

printf("SharedVariable\t %8x\t %8x\t %s\n",

SharedVariable,

pSharedVariable,

GetPageAttributesString(

aSharedVariable));

printf("NonSharedVariable\t %8x\t %8x\t %s\n",

NonSharedVariable,

pNonSharedVariable,

GetPageAttributesString(

aNonSharedVariable));

printf("------------------------------------\n\n");

return 0;

}

The DisplayVirtualAndPhysicalAddresses() function is a utility function that displays the virtual address, the physical address, and the page attributes for different memory sections. It uses the global pointers to the different sections in MYDLL.DLL initialized by the LoadDllAndInitializeVirtualAddresses() function. It uses the GetPhysicalAddressAndPageAttributes() function to get hold of the physical page address and the page attributes for the given virtual address. The first parameter to the GetPhysicalAddressAndPageAttributes() function is the input virtual address. The function fills in the physical address for the input virtual address in the memory location pointed to by the second parameter and the page attributes in the location pointed to by the third parameter.

int FirstInstance()

{

printf("***This is the first instance of the"

" showphys program***\n\n");

printf("Loading DLL MYDLL.DLL\n");

if (LoadDllAndInitializeVirtualAddresses()!=0) {

return -1;

}

DisplayVirtualAndPhysicalAddresses();

printf("Now Run another copy of showphys ...\n");

getchar();

FreeDll();

}

Memory Management -- Part 5

2007-11-22T11:31:00.001-08:00

Memory Management

The first two members dictate the address range represented by the VAD node. Each VAD tree node maintains a pointer to the parent node and a pointer to the left child and the right child. The VAD tree is a binary tree. For every node in the tree, the left subtree consists of nodes representing lower address ranges, and the right subtree consists of nodes representing the higher address ranges. The last member in the VAD node is the flags for the address range.

The VADDUMP.C file has a few other global variables apart from the VadInfoArray. A couple of global variables are used while locating the root of the VAD tree. The PEB of a process points to the VAD tree root for that process. The offset of this pointer inside the PEB varies with the Windows NT version. We set the VadRootOffset to the appropriate offset value of the VAD root pointer depending on the Windows NT version. There is a similar problem of Windows NT version dependency while accessing the PEB for the process. We use the Thread Environment Block (TEB) to get to the PEB. One field in TEB points to the PEB, but the offset of this field inside the TEB structure varies with the Windows NT version. We set the PebOffset variable to the appropriate offset value of the PEB pointer inside the TEB structure depending on the Windows NT version. Another global variable, NtVersion, stores the version of Windows NT running on the machine.

That leaves us with two more global variables, namely, VadInfoArrayIndex and VadTreeRoot. The VadInfoArrayIndex is the number of initialized entries in the VadInfoArray. The VadInfoArray entries after VadInfoArrayIndex are free. The VadTreeRoot variable stores the root of the VAD tree.

The sample has been tested on Windows NT 3.51, 4.0 and Windows 2000 beta2. The sample will run on other versions of Windows 2000, provided the offsets of VadRoot and PEB remain same.

/* Recursive function which walks the vad tree and

* fills up the global VadInfoArray with the Vad

* entries. Function is limited by the

* MAX_VAD_ENTRIES. Other VADs after this are not

* stored

*/

void _stdcall VadTreeWalk(PVAD VadNode)

{

if (VadNode == NULL) {

return;

}

if (VadInfoArrayIndex >= MAX_VAD_ENTRIES) {

return;

}

VadTreeWalk(VadNode->LeftLink);

VadInfoArray[VadInfoArrayIndex].VadLocation = VadNode;

VadInfoArray[VadInfoArrayIndex].Vad.StartingAddress =

VadNode->StartingAddress;

VadInfoArray[VadInfoArrayIndex].Vad.EndingAddress =

VadNode->EndingAddress;

if (NtVersion == 5) {

(DWORD)VadInfoArray[VadInfoArrayIndex].

Vad.StartingAddress <<= 12;

(DWORD)VadInfoArray[VadInfoArrayIndex].

Vad.EndingAddress += 1;

(DWORD)VadInfoArray[VadInfoArrayIndex].

Vad.EndingAddress <<= 12;

(DWORD)VadInfoArray[VadInfoArrayIndex].

Vad.EndingAddress -= 1;

}

VadInfoArray[VadInfoArrayIndex].Vad.ParentLink =

VadNode->ParentLink;

VadInfoArray[VadInfoArrayIndex].Vad.LeftLink =

VadNode->LeftLink;

VadInfoArray[VadInfoArrayIndex].Vad.RightLink =

VadNode->RightLink;

VadInfoArray[VadInfoArrayIndex].Vad.Flags =

VadNode->Flags;

VadInfoArrayIndex++;

VadTreeWalk(VadNode->RightLink);

}

The VadTreeWalk() function is executed in the kernel mode using the callgate mechanism. The function traverses the VAD tree in the in-order fashion and fills up the VadInfoArray. The function simply returns if the node pointer parameter is NULL or the VadInfoArray is full. Otherwise, the function recursively calls itself for the left subtree. The recursion is terminated when the left child pointer is NULL. The function then fills up the next free entry in the VadInfoArray and increments the VadInfoArrayIndex to point to the next free entry. Windows 2000 stores the page numbers instead of the actual addresses in VAD. Hence, for Windows 2000, we need to calculate the starting address and the ending address from the page numbers stored in these fields. As the last step in the in-order traversal, the function issues a self-recursive to process the right subtree.

/* C function called through assembly stub */

void _stdcall CFuncDumpVad(PVAD VadRoot)

{

VadTreeRoot = VadRoot;

VadInfoArrayIndex = 0;

VadTreeWalk(VadRoot);

}

The CfuncDumpVad is the caller of the VadTreeWalk() function. It just initializes the global variables used by the VadTreeWalk() function and calls the VadTreeWalk() function for the root of the VAD tree.

/* Displays the Vad tree */

void VadTreeDisplay()

{

int i;

printf("VadRoot is located @%08x\n\n",

VadTreeRoot);

printf("Vad@\t Starting\t Ending\t Parent\t "

"LeftLink\t RightLink\n");

for (i=0; i < VadInfoArrayIndex; i++) {

printf("%08x %08x %08x %8x %08x %08x\n",

VadInfoArray[i].VadLocation,

VadInfoArray[i].Vad.StartingAddress,

VadInfoArray[i].Vad.EndingAddress,

VadInfoArray[i].Vad.ParentLink,

VadInfoArray[i].Vad.LeftLink,

VadInfoArray[i].Vad.RightLink);

}

printf("\n\n");

}

The VadTreeDisplay() function is a very simple function that is executed in user mode. The function iterates through all the entries initialized by the VadTreeWalk() function and prints the entries. Essentially, the function prints the VAD tree in the infix order because the VadTreeWalk() function dumps the VAD tree in the infix order.

void SetDataStructureOffsets()

{

switch (NtVersion) {

case 3:

PebOffset = 0x40;

VadRootOffset = 0x170;

break;

case 4:

PebOffset = 0x44;

VadRootOffset = 0x170;

break;

case 5:

PebOffset = 0x44;

VadRootOffset = 0x194;

break;

}

}

As we described earlier, the offset of the PEB pointer within TEB and the offset of the VAD root pointer within the PEB are dependent on the Windows NT version. The SetDataStructureOffsets() function sets the global variables indicating these offsets depending on the Windows NT version.

main()

{

WORD CallGateSelector;

int rc;

short farcall[3];

void DumpVad(void);

void *ptr;

OSVERSIONINFO VersionInfo;

VersionInfo.dwOSVersionInfoSize = sizeof(VersionInfo);

if (GetVersionEx(&VersionInfo) == TRUE) {

NtVersion = VersionInfo.dwMajorVersion;

}

if ((NtVersion < 3)||(NtVersion > 5)) {

printf("Unsupported NT version, exiting...");

return 0;

}

SetDataStructureOffsets();

/* Creates call gate to read vad tree from Ring 3

*/

rc = CreateCallGate(DumpVad, 0, &CallGateSelector);

if (rc != SUCCESS) {

printf("CreateCallGate failed, rc=%x\n", rc);

return 1;

}

farcall[2] = CallGateSelector;

_asm {

call fword ptr [farcall]

}

printf("Dumping the Vad tree ...\n\n");

VadTreeDisplay();

printf("Allocating memory using VirtualAlloc");

ptr = VirtualAlloc(NULL, 4096, MEM_COMMIT,

PAGE_READONLY);

if (ptr == NULL) {

printf("Unable to allocate memory\n");

goto Quit;

}

printf("\nMemory allocated @%x\n", ptr);

_asm {

call fword ptr [farcall]

}

printf("\n\nDumping the Vad tree again...\n\n");

VadTreeDisplay();

Quit:

rc = FreeCallGate(CallGateSelector);

if (rc != SUCCESS) {

printf("FreeCallGate failed, Selector=%x, rc=%x\n",

CallGateSelector, rc);

}

return 0;

}

The main() function starts by getting the Windows NT version and calling SetDataStructureOffsets() to set the global variables storing the offsets for the PEB and the VAD tree root. It then creates a callgate in the same manner as in the SHOWDIR sample program. Issuing a call through this callgate ultimately results in the execution of the VadTreeWalk() function that fills up the VadInfoArray. The main() function then calls the VadTreeDisplay() function to print the VadInfoArray entries.

We also show you the change in the VAD tree due to memory allocation in this sample program. After printing the VAD tree once, the program allocates a chunk of memory. Then, the program issues the callgate call again and prints the VAD tree after returning from the call. You can observe the updates that happened to the VAD tree because of the memory allocation. The program frees up the callgate before exiting.

Listing 4-6: RING0.ASM

.386

.model small

.code

public _DumpVad

extrn _CFuncDumpVad@4:near

extrn _PebOffset:near

extrn _VadRootOffset:near

include ..\include\undocnt.inc

_DumpVad proc

Ring0Prolog

;Gets the current thread

MOV EAX,FS:[00000124h]

;Gets the current process

ADD EAX, DWORD PTR [_PebOffset]

MOV EAX,[EAX]

;Push Vad Tree root

ADD EAX, DWORD PTR [_VadRootOffset]

MOV EAX, [EAX]

PUSH EAX

CALL _CFuncDumpVad@4

Ring0Epilog

RETF

_DumpVad endp

END

The function to be called from the callgate needs to be written in the Assembly language for reasons already described. The DumpVad() function gets hold of the VAD root pointer and calls the CFuncDumpVad() function that dumps the VAD tree in the VadInfoArray. The function gets hold of the VAD root from the PEB after getting hold of the PEB from the TEB. The TEB of the currently executing thread is always pointed to by FS:128h. As described earlier, the offset of the VAD root pointer inside PEB and the offset of the PEB pointer inside the TEB vary with the Windows NT version. The DumpVad() function uses the offset values stored in the global variable by the SetDataStructureOffsets() function.

Listing 4-7 presents the output from an invocation of the VADDUMP program. Note that the VAD tree printed after allocating memory at address 0x300000 shows an additional entry for that address range.

Listing 4-7: Program output

Dumping the Vad tree...

VadRoot is located @fe21a9c8

Vad@ Starting Ending Parent LeftLink RightLink

fe216b08 00010000 00010fff fe21a9c8 00000000 fe25a0e8

fe25a0e8 00020000 00020fff fe216b08 00000000 fe275da8

fe275da8 00030000 0012ffff fe25a0e8 00000000 fe22a428

fe22a428 00130000 00130fff fe275da8 00000000 fe26b328

fe26b328 00140000 0023ffff fe22a428 00000000 fe210fc8

fe210fc8 00240000 0024ffff fe26b328 00000000 fe21a8c8

fe21a8c8 00250000 00258fff fe210fc8 00000000 fe21be68

fe21be68 00260000 0026dfff fe21a8c8 00000000 fe215dc8

fe215dc8 00270000 002b0fff fe21be68 00000000 fe231e88

fe231e88 002c0000 002c0fff fe215dc8 00000000 fe2449e8

fe2449e8 002d0000 002dffff fe231e88 00000000 fe21cb48

fe21cb48 002e0000 002e0fff fe2449e8 00000000 fe23b7a8

fe23b7a8 002f0000 002fffff fe21cb48 00000000 00000000

fe21a9c8 00400000 0040cfff 0 fe216b08 fe23c488

fe21b3e8 10000000 1000dfff fe2333e8 00000000 fe226348

fe2176c8 77e20000 77e4bfff fe226348 00000000 fe2326e8

fe2152c8 77e50000 77e54fff fe2326e8 00000000 00000000

fe2326e8 77e60000 77e9bfff fe2176c8 fe2152c8 00000000

fe226348 77ea0000 77ed7fff fe21b3e8 fe2176c8 fe2197c8

fe2197c8 77ee0000 77f12fff fe226348 00000000 00000000

fe2333e8 77f20000 77f73fff fe23c488 fe21b3e8 00000000

fe23c488 77f80000 77fcdfff fe21a9c8 fe2333e8 fe25aa88

fe22b408 7f2d0000 7f5cffff fe25aa88 00000000 fe22c4a8

fe22c4a8 7f5f0000 7f7effff fe22b408 00000000 fe23f5e8

fe23f5e8 7ff70000 7ffaffff fe22c4a8 00000000 00000000

fe25aa88 7ffb0000 7ffd3fff fe23c488 fe22b408 fe218288

fe21da88 7ffde000 7ffdefff fe218288 00000000 00000000

fe218288 7ffdf000 7ffdffff fe25aa88 fe21da88 00000000

Allocating memory using VirtualAlloc

Memory allocated @300000

Dumping the Vad tree again...

VadRoot is located @fe21a9c8

Vad@ Starting Ending Parent LeftLink RightLink

fe216b08 00010000 00010fff fe21a9c8 00000000 fe25a0e8

fe25a0e8 00020000 00020fff fe216b08 00000000 fe275da8

fe275da8 00030000 0012ffff fe25a0e8 00000000 fe22a428

fe22a428 00130000 00130fff fe275da8 00000000 fe26b328

fe26b328 00140000 0023ffff fe22a428 00000000 fe210fc8

fe210fc8 00240000 0024ffff fe26b328 00000000 fe21a8c8

fe21a8c8 00250000 00258fff fe210fc8 00000000 fe21be68

fe21be68 00260000 0026dfff fe21a8c8 00000000 fe215dc8

fe215dc8 00270000 002b0fff fe21be68 00000000 fe231e88

fe231e88 002c0000 002c0fff fe215dc8 00000000 fe2449e8

fe2449e8 002d0000 002dffff fe231e88 00000000 fe21cb48

fe21cb48 002e0000 002e0fff fe2449e8 00000000 fe23b7a8

fe23b7a8 002f0000 002fffff fe21cb48 00000000 fe27b628

fe27b628 00300000 00300fff fe23b7a8 00000000 00000000

fe21a9c8 00400000 0040cfff 0 fe216b08 fe23c488

fe21b3e8 10000000 1000dfff fe2333e8 00000000 fe226348

fe2176c8 77e20000 77e4bfff fe226348 00000000 fe2326e8

fe2152c8 77e50000 77e54fff fe2326e8 00000000 00000000

fe2326e8 77e60000 77e9bfff fe2176c8 fe2152c8 00000000

fe226348 77ea0000 77ed7fff fe21b3e8 fe2176c8 fe2197c8

fe2197c8 77ee0000 77f12fff fe226348 00000000 00000000

fe2333e8 77f20000 77f73fff fe23c488 fe21b3e8 00000000

fe23c488 77f80000 77fcdfff fe21a9c8 fe2333e8 fe25aa88

fe22b408 7f2d0000 7f5cffff fe25aa88 00000000 fe22c4a8

fe22c4a8 7f5f0000 7f7effff fe22b408 00000000 fe23f5e8

fe23f5e8 7ff70000 7ffaffff fe22c4a8 00000000 00000000

fe25aa88 7ffb0000 7ffd3fff fe23c488 fe22b408 fe218288

fe21da88 7ffde000 7ffdefff fe218288 00000000 00000000

fe218288 7ffdf000 7ffdffff fe25aa88 fe21da88 00000000

The output of the VADDUMP program does not really look like a tree. You have to trace through the output to get the tree structure. The entry with a null parent link is the root of the tree. Once you find the root, you can follow the child pointers. To follow a child pointer, search the pointer in the first column, named Vad@, in the output. The Vad entry with the same Vad@ is the entry for the child that you are looking for. An all-zero entry for a left/right child pointer indicates that there is no left/right subtree for the node. Figure 4-5 shows a partial tree constructed from the output shown previously.

Memory Management -- Part 3

2007-11-22T11:29:00.000-08:00


Memory Management

Let’s analyze, one step at a time, why the two entries are different. The page tables themselves need to be mapped onto some linear address. When Windows NT needs to access the page tables, it uses this linear address range. To represent 4GB of memory divided into 1MB pages of 4K each, we need 1K page tables each having 1K entries. To map these 1K page tables, Windows NT reserves 4MB of linear address space in each process. As we saw earlier, each process has a different set of page tables. Whatever the process, Windows NT maps the page tables on the linear address range from 0xC0000000 to 0xC03FFFFF. Let’s call this linear address range as the page table address range. In other words, the page table address range maps to different page tables–that is, to different physical pages–for different processes. As you may have noticed, the page table addresses range falls in the kernel address space. Windows NT cannot map this crucial system data structure in the user address space and allow user-mode processes to play with the memory. Ultimately, the result is that two processes cannot share pages in the page table address range although the addresses lie in the kernel-mode address range.

Exactly one page table is required to map 4MB address space because each page table has 1K entries and each entry corresponds to a 4K page. Consequently, Windows NT cannot share the page table corresponding to the page table address range. This accounts for one of the two mysterious entries in the page table directory. However, the entry’s mystery does not end here–there is one more subtle twist to this story. The physical address specified in this entry matches the physical address of the page table directory. The obvious conclusion is that the page table directory acts also as the page table for the page table address range. This is possible because the formats of the page table directory entry and PTE are the same on 80386.

The processor carries out an interesting sequence of actions when the linear address within the page table address range is translated to a physical address. Let’s say that the CR3 register points to page X. As the first step in the address translation process, the processor treats the page X as the page table directory and finds out the page table for the given linear address. The page table happens to be page X again. The processor now treats page X as the required page table and finds out the physical address from it. A more interesting case occurs when the operating system is accessing the page table directory itself. In this case, the physical address also falls in page X!

Let’s now turn to the second mysterious entry. The 4MB area covered by this page directory entry is internally referred to as hyperspace. This area is used for mapping the physical pages belonging to other processes into virtual address space. For example, a function such as MmMapPageInHyperspace() uses the virtual addresses in this range. This area is also used during the early stages of process creation. For example, when a parent process such as PROGMAN.EXE spawns a child process such as NOTEPAD.EXE, PROGMAN.EXE has to create the address space for NOTEPAD.EXE. This is done as a part of the MmCreateProcessAddressSpace() function. For starting any process, an address space must be created for the process. Address space is nothing but page directory. Also, the upper-half entries of page directory are common for all processes except for the two entries that we have already discussed. These entries need to be created for the process being spawned. The MmCreateProcessAddressSpace() function allocates three pages of memory: the first page for the page directory, the second page for holding the hyperspace page table entries, and the third page for holding the working set information for the process being spawned.

Once these pages are allocated, the function maps the first physical page in the address space using the MmMapPageInHyperSpace() function. Note that the MmMapPageInHyperSpace() function runs in the context of PROGMAN.EXE. Now the function copies the page directory entries in the upper half of the page directory to the mapped hyperspace virtual address. In short, PROGMAN.EXE creates the page directory for the NOTEPAD.EXE.

Windows NT supports memory-mapped files. When two processes map the same file, they share the same set of physical pages. Hence, memory-mapped files can be used for sharing memory. In fact, Windows NT itself uses memory-mapped files to load DLLs and executables. If two processes map the same DLL, they automatically share the DLL pages. The memory-mapped files are implemented using the section object under Windows NT. A data structure called PROTOPTE is associated with each section object. This data structure is a variable-length structure based on the size of the section. This data structure contains a 4-byte entry for each page in the virtual address space mapped by the section object. Each 4-byte entry has the same structure as that of the PTE. When the page is not being used by any of the processes, the protopte entry is invalid and contains enough information to get the page back. In this case, the CPU PTE contains a fixed value that is 0xFFFFF480, which indicates that accessing this page will be considered a protopte fault.

Now comes the toughest of all questions: "How can Windows NT give away 4GB of memory to each process when there is far less physical RAM available on the board?" Windows NT, as well as all other operating systems that allow more address space than actual physical memory, uses a technique called virtual memory to achieve this. In the next section, we discuss virtual memory management in Windows NT.


VIRTUAL MEMORY MANAGEMENT

The basic idea behind virtual memory is very simple. For each process, the operating system maps few addresses to real physical memory because RAM is expensive and relatively rare. Remaining memory for each process is really maintained on secondary storage (usually a hard disk). That’s why it is called virtual memory. The addresses that are not mapped on physical RAM are marked as such. Whenever a process accesses such an address, the operating system brings the data into memory from secondary storage. If the operating system runs out of physical RAM, some data is thrown out to make space. We can always get back this data because a copy is maintained on secondary storage. The data to be thrown out is decided by the replacement policy. Windows NT uses First-In-First-Out (FIFO) replacement policy. According to this policy, the oldest data (that is, the data that was brought in the RAM first) is thrown out whenever there is a space crunch.

To implement virtual memory management, Windows NT needs to maintain a lot of data. First, it needs to maintain whether each address is mapped to physical RAM or the data is to be brought in from secondary storage when a request with the address comes. Maintaining this information for each byte itself takes a lot of space (actually, more space than the address space for which the information is to be maintained). So Windows NT breaks the address space into 4KB pages and maintains this information in page tables. As we saw earlier, a page table entry (PTE) consists of the address of the physical page (if the page is mapped to physical RAM) and attributes of the page. Since the processor heavily depends on PTEs for address translation, the structure of PTE is processor dependent.

If a page is not mapped onto physical RAM, Windows NT marks the page as invalid. Any access to this page causes a page fault, and the page fault handler can bring in the page from the secondary storage. To be more specific, when the page contains DLL code or executable module code, the page is brought in from the DLL or executable file. When the page contains data, it is brought in from the swap file. When the page represents a memory-mapped file area, it is brought in from the corresponding file. Windows NT needs to keep track of free physical RAM so that it can allocate space for a page brought in from secondary storage in case of a page fault. This information is maintained in a kernel data structure called the Page Frame Database (PFD). The PFD also maintains a FIFO list of in-memory pages so that it can decide on pages to throw out in case of a space crunch.

Before throwing out a page, Windows NT must ensure that the page is not dirty. Otherwise, it needs to write that page to secondary storage before throwing it out. If the page is not shared, the PFD contains the pointer to PTE so that if the operating system decides to throw out a particular page, it can then go back and mark the PTE as invalid. If the page is shared, the PFD contains a pointer to the corresponding PROTOPTE entry. In this case, the PFD also contains a reference count for the page. A page can be thrown out only if its reference count is 0. In general, the PFD maintains the status of every physical page.

The PFD is an array of 24-byte entries, one for each physical page. Hence, the size of this array is equal to the number of physical pages that are stored in a kernel variable, namely, MmNumberOfPhysicalPages. The pointer to this array is stored in a kernel variable, namely, MmpfnDatabase. A physical page can be in several states–for example, it can be in-use, free, free but dirty, and so on. A PFD entry is linked in a doubly linked list, depending on the state of the physical page represented by it. For example, the PFD entry representing a free page is linked in the free pages list. Figure 4-4 shows these lists linked through the PFD. The forward links are shown on the left side of the PFD, and the backward links are shown on the right side.

There are in all six kinds of lists. The heads of these lists are stored in following kernel variables:

MmStandbyPageListHead

MmModifiedNoWritePageListHead

MmModifiedPageListHead

MmFreePageListHead

MmBadPageListHead

MmZeroedPageListHead

All these list heads are actually structures of 16 bytes each. Here is the structure definition:

typedef struct PageListHead {

 DWORD NumberOfPagesInList,

 DWORD TypeOfList,

 DWORD FirstPage,

 DWORD LastPage

} PageListHead_t;

The FirstPage field can be used as an index into the PFD. The PFD entry contains a pointer to the next page. Using this, you can traverse any of the lists. Here is the structure definition for the PFD entry:

typedef struct PfdEntry {

 DWORD NextPage,

 void *PteEntry/*PpteEntry,

 DWORD PrevPage,

 DWORD PteReferenceCount,

 void *OriginalPte,

 DWORD Flags;

} PfdEntry_t;

Using this, you can easily write a program to dump the PFD. However, there is one problem: kernel variables, such as list heads, MmPfnDatabase, and MmNumberOfPhysicalPages, are not exported. Therefore, you have to deal with absolute addresses, which makes the program dependent on the Windows NT version and build type.


VIRTUAL ADDRESS DESCRIPTORS

Along with the free physical pages, Windows NT also needs to keep track of the virtual address space allocation for each process. Whenever a process allocates a memory block–for example, to load a DLL–Windows NT checks for a free block in the virtual address space, allocates virtual address space, and updates the virtual address map accordingly. The most obvious place to maintain this information is page tables. For each process, Windows NT maintains separate page tables. There are 1 million pages, and each page table entry is 4 bytes. Hence, full page tables for a single process would take 4MB of RAM! There is a solution to this: Page tables themselves can be swapped out. It is inefficient to swap in entire page tables when a process wants to allocate memory. Hence, Windows NT maintains a separate binary search tree containing the information about current virtual space allocation for each process. A node in this binary search tree is called a Virtual Address Descriptor (VAD). For each block of memory allocated to a process, Windows NT adds a VAD entry to the binary search tree. Each VAD entry contains the allocated address range–that is, the start address and the end address of the allocated block, pointers to left and right children VADs, and a pointer to the parent VAD. The process environment block (PEB) contains a pointer, namely, VadRoot, to the root of this tree.

Listing 4-5: VADDUMP.C

/* Should be compiled in release mode */

#define _X86_




#include 

#include 

#include 




#include "undocnt.h"

#include "gate.h"




/*Define the WIN32 calls we are using, since we can not include both

NTDDK.H and

WINDOWS.H in the same ’C’ file.*/

typedef struct _OSVERSIONINFO{

 ULONG dwOSVersionInfoSize;

 ULONG dwMajorVersion;

 ULONG dwMinorVersion;

 ULONG dwBuildNumber;

 ULONG dwPlatformId;

 CCHAR szCSDVersion[ 128 ];

} OSVERSIONINFO, *LPOSVERSIONINFO;

BOOLEAN _stdcall GetVersionExA(LPOSVERSIONINFO);

PVOID _stdcall VirtualAlloc(PVOID, ULONG, ULONG, ULONG);




/* Max vad entries */

#define MAX_VAD_ENTRIES   0x200




/* Following variables are accessed in RING0.ASM */

ULONG NtVersion;

ULONG PebOffset;

ULONG VadRootOffset;




#pragma pack(1)

typedef struct VadInfo {

 void *VadLocation;

 VAD Vad;

} VADINFO, *PVADINFO;

#pragma pack()




VADINFO VadInfoArray[MAX_VAD_ENTRIES];

int VadInfoArrayIndex;

PVAD VadTreeRoot;

The initial portion of the VADDUMP.C file has a few definitions apart from the header inclusion. In this program, we use the callgate mechanism as we did in the showdir program–hence the inclusion of the GATE.H header file. After the header inclusion, the file defines the maximum number of VAD entries that we’ll process. There is no limit on the nodes in a VAD tree. We use the callgate mechanism for kernel-mode execution of a function that dumps the VAD tree in an array accessible from the user mode. This array can hold up to MAX_VAD_ENTRIES entries. Each entry in the array is of type VADINFO. The VADINFO structure has two members: the address of the VAD tree node and the actual VAD tree node. The VAD tree node structure is defined in the UNDOCNT.H file as follows:

typedef struct vad {

void *StartingAddress;

void *EndingAddress;

struct vad *ParentLink;

struct vad *LeftLink;

struct vad *RightLink;

DWORD Flags;

}VAD, *PVAD;

Memory Management--Part 2

2007-11-22T11:28:00.000-08:00

Memory Management

Process Isolation
The next question that comes to mind is, “How does Windows NT keep processes from seeing each other’s address space?” Again, the mechanism for achieving this design goal is simple. Windows NT maintains a separate page table directory for each process and based on the process in execution, it switches to the corresponding page table directory. As the page table directories for different processes point to different page tables and these page tables point to different physical pages and only one directory is active at a time, no process can see any other process’s memory. When Windows NT switches the execution context, it also sets the CR3 register to point to the appropriate page table directory. The kernel-mode address space is mapped for all processes, and all page table directories have entries for kernel address space. However, another feature of 80386 is used to disallow user-mode code from accessing kernel address space. All the kernel pages are marked as supervisor pages; therefore, user-mode code cannot access them.

Code Page Sharing in DLLs
For sharing code pages of a DLL, Windows NT maps corresponding page table entries for all processes sharing the DLL onto the same set of physical pages. For example, if process A loads X.DLL at address xxxx and process B loads the same X.DLL at address yyyy, then the PTE for xxxx in process A’s page table and the PTE for yyyy in process B’s page table point to the same physical page. Figure 4-2 shows two processes sharing a page via same page table entries. The DLL pages are marked as read-only so that a process inadvertently attempting to write to this area will not cause other processes to crash.

Note: This is guaranteed to be the case when xxxx==yyyy. However, if xxxx!=yyyy, the physical page might not be same. We will discuss the reason behind this later in the chapter.

Kernel address space is shared using a similar technique. Because the entire kernel space is common for all processes, Windows NT can share page tables directly. Figure 4-3 shows how processes share physical pages by using same page tables. Consequently, the upper half of the page table directory entries are the same for all processes.

Listing 4-1 shows the sample program that demonstrates this.

Listing 4-1: SHOWDIR.C

/* Should be compiled in release mode to run properly */

#include

#include

#include

#include "gate.h"

/* Global array to hold the page directory */

DWORD PageDirectory[1024];

This initial portion of the SHOWDIR.C file contains, apart from the header inclusion, the global definition for the array to hold the page directory. The inclusion of the header file GATE.H is of interest. This header file prototypes the functions for using the callgate mechanism. Using the callgate mechanism, you can execute your code in the kernel mode without writing a new device driver.

XREF: We discuss the callgate mechanism in Chapter 10.

For this sample program, we need this mechanism because the page directory is not accessible to the user-mode code. For now, it’s sufficient to know that the mechanism allows a function inside a normal executable to be executed in kernel mode. Turning on to the definition of the page directory, we have already described that the size of each directory entry is 4 bytes and a page directory contains 1024 entries. Hence, the PageDirectory is an array of 1024 DWORDs. Each DWORD in the array represents the corresponding directory entry.

/* C function called from the assembly stub */

void _stdcall CFuncGetPageDirectory()

{

DWORD *PageDir=(DWORD *)0xC0300000;

int i=0;

for (i=0; i<1024; i++) {

PageDirectory[i] = PageDir[i];

}

}

CfuncGetPageDirectory() is the function that is executed in the kernel mode using the callgate mechanism. This function simply makes a copy of the page directory in the user-mode memory area so that the other user-mode code parts in the program can access it. The page directory is mapped at virtual address 0xC0300000 in every process’s address space. This address is not accessible from the user mode. The CFuncGetPageDirectory() function copies 1024 DWORDs from the 0xC0300000 address to the global PageDirectory variable that is accessible to the user-mode code in the program.

/* Displays the contents of page directory. Starting

* virtual address represented by the page directory

* entry is shown followed by the physical page

* address of the page table

*/

void DisplayPageDirectory()

{

int i;

int ctr=0;

printf("Page directory for the process, pid=%x\n",

GetCurrentProcessId());

for (i=0; i<1024; i++) {

if (PageDirectory[i]&0x01) {

if ((ctr%3)==0) {

printf("\n");

}

printf("%08x:%08x ", i << 22,

PageDirectory[i] & 0xFFFFF000);

ctr++;

}

}

printf("\n");

}

The DisplayPageDirectory() function operates in user mode and prints the PageDirectory array that is initialized by the CfuncGetPageDirectory() function. The function checks the Least Significant Bit (LSB) of each of the entries. A page directory entry is valid only if the last bit or the LSB is set. The function skips printing invalid entries. The function prints three entries on every line or, in other words, prints a newline character for every third entry. Each directory entry is printed as the logical address and the address of the corresponding page table as obtained from the page directory. As described earlier, the first 10 bits (or the 10 Most Significant Bits [MSB]) of the logical address are used as an index in the page directory. In other words, a directory entry at index i represents the logical addresses that have i as the first 10 bits. The function prints the base of the logical address range for each directory entry. The base address (that is, the least address in the range) has the last 22 bits (or 22 LSBs) as zeros. The function obtains this base address by shifting i to the first 10 bits. The address of the page table corresponding to the logical address is stored in the first 20 bits (or 20 MSBs) of the page directory entry. The 12 LSBs are the flags for the entry. The function calculates the page table address by masking off the flag bits.

main()

{

WORD CallGateSelector;

int rc;

static short farcall[3];

/* Assembly stub that is called through callgate */

extern void GetPageDirectory(void);

/* Creates a callgate to read the page directory

* from Ring 3 */

rc = CreateCallGate(GetPageDirectory, 0,

&CallGateSelector);

if (rc == SUCCESS) {

farcall[2] = CallGateSelector;

_asm {

call fword ptr [farcall]

}

DisplayPageDirectory();

getchar();

/* Releases the callgate */

rc=FreeCallGate(CallGateSelector);

if (rc!=SUCCESS) {

printf("FreeCallGate failed, "

"CallGateSelector=%x, rc=%x\n",

CallGateSelector, rc);

}

} else {

printf("CreateCallGate failed, rc=%x\n", rc);

}

return 0;

}

The main() function starts by creating a callgate that sets up the GetPageDirectory() function to be executed in the kernel mode. The GetPageDirectory() function is written in Assembly language and is a part of the RING0.ASM file. The CreateCallGate() function, used by the program to create the callgate, is provided by CALLGATE.DLL. The function returns with a callgate selector.

XREF: The mechanism of calling the desired function through callgate is explained in Chapter 10.

We’ll quickly mention a few important points here. The callgate selector returned by CreateCallGate() is a segment selector for the given function: in this case, GetPageDirectory(). To invoke the function pointed by the callgate selector, you need to issue a far call instruction. The far call instruction expects a 16-bit segment selector and a 32-bit offset within the segment. When you are calling through a callgate, the offset does not matter; the processor always jumps at the start of the function pointed to by the callgate. Hence, the program only initializes the third member of the farcall array that corresponds to the segment selector. Issuing a call through the callgate transfers the execution control to the GetPageDirectory() function. This function calls the CfuncGetPageDirectory() function that copies the page directory in the PageDirectory array. After the callgate call returns, the program prints the page directory copied in the PageDirectory by calling the DisplayPageDirectory() function. The program frees the callgate before exiting.

Listing 4-2: RING0.ASM

.386

.model small

.code

include ..\include\undocnt.inc

public _GetPageDirectory

extrn _CFuncGetPageDirectory@0:near

;Assembly stub called from callgate

_GetPageDirectory proc

Ring0Prolog

call _CFuncGetPageDirectory@0

Ring0Epilog

retf

_GetPageDirectory endp

END

The function to be called from the callgate needs to be written in assembly language for a couple of reasons. First, the function needs to execute a prolog and an epilog, both of which are assembly macros, to allow paging in kernel mode. Second, the function needs to issue a far return at the end. The function leaves the rest of the job to the CFuncGetPageDirectory() function written in C.

If you compare the output of the showdir program for two different processes, you find that the upper half of the page table directories for the two processes is exactly the same except for two entries. In other words, the corresponding kernel address space for these two entries is not shared by the two processes.

Listing 4-3: First instance of SHOWDIR

Page directory for the process, pid=6f

00000000:01026000 00400000:00f65000 10000000:0152f000

5f800000:00e46000 77c00000:0076b000 7f400000:012cb000

7fc00000:0007e000 80000000:00000000 80400000:00400000

80800000:00800000 80c00000:00c00000 81000000:01000000

81400000:01400000 81800000:01800000 81c00000:01c00000

82000000:02000000 82400000:02400000 82800000:02800000

82c00000:02c00000 83000000:03000000 83400000:03400000

83800000:03800000 83c00000:03c00000 84000000:04000000

84400000:04400000 84800000:04800000 84c00000:04c00000

85000000:05000000 85400000:05400000 85800000:05800000

85c00000:05c00000 86000000:06000000 86400000:06400000

86800000:06800000 86c00000:06c00000 87000000:07000000

87400000:07400000 87800000:07800000 87c00000:07c00000

a0000000:0153d000 c0000000:00e5d000 c0400000:00c9e000

c0c00000:00041000 c1000000:00042000 c1400000:00043000

c1800000:00044000 c1c00000:00045000 c2000000:00046000

c2400000:00047000 c2800000:00048000 c2c00000:00049000

c3000000:0004a000 c3400000:0004b000 c3800000:0004c000

c3c00000:0004d000 c4000000:0004e000 c4400000:0000f000

c4800000:00050000 c4c00000:00051000 c5000000:00052000

c5400000:00053000 c5800000:00054000 c5c00000:00055000

c6000000:00056000 c6400000:00057000 c6800000:00058000

c6c00000:00059000 c7000000:0005a000 c7400000:0005b000

c7800000:0005c000 c7c00000:0005d000 c8000000:0005e000

c8400000:0005f000 c8800000:00020000 c8c00000:00021000

c9000000:00022000 c9400000:00023000 c9800000:00024000

c9c00000:00025000 ca000000:00026000 ca400000:00027000

ca800000:00028000 cac00000:00029000 cb000000:0002a000

cb400000:0002b000 cb800000:0002c000 cbc00000:0002d000

cc000000:0002e000 cc400000:0002f000 cc800000:002f0000

ccc00000:002f1000 cd000000:002f2000 cd400000:002f3000

cd800000:002f4000 cdc00000:002f5000 ce000000:002f6000

ce400000:00037000 ce800000:00038000 cec00000:00039000

cf000000:0003a000 cf400000:0003b000 cf800000:0003c000

cfc00000:0003d000 d0000000:0003e000 d0400000:0003f000

d0800000:00380000 d0c00000:00301000 d1000000:00302000

d1400000:00303000 d1800000:00304000 d1c00000:00305000

d2000000:00306000 d2400000:00307000 d2800000:00308000

d2c00000:00309000 d3000000:0030a000 d3400000:0030b000

d3800000:0030c000 d3c00000:0030d000 d4000000:0030e000

d4400000:0004f000 d4800000:00310000 d4c00000:00311000

e1000000:00315000 e1400000:010fe000 fc400000:0038d000

fc800000:0038e000 fcc00000:0038f000 fd000000:00390000

fd400000:00391000 fd800000:00392000 fdc00000:00393000

fe000000:00394000 fe400000:00395000 fe800000:00396000

fec00000:00397000 ff000000:00398000 ff400000:00399000

ff800000:0039a000 ffc00000:00031000

Listing 4-4: Second instance of SHOWDIR

Page directory for the process, pid=7d

00000000:00fa1000 00400000:00fa0000 10000000:0110a000

5f800000:015ac000 77c00000:01a73000 7f400000:013ac000

7fc00000:0145e000 80000000:00000000 80400000:00400000

80800000:00800000 80c00000:00c00000 81000000:01000000

81400000:01400000 81800000:01800000 81c00000:01c00000

82000000:02000000 82400000:02400000 82800000:02800000

82c00000:02c00000 83000000:03000000 83400000:03400000

83800000:03800000 83c00000:03c00000 84000000:04000000

84400000:04400000 84800000:04800000 84c00000:04c00000

85000000:05000000 85400000:05400000 85800000:05800000

85c00000:05c00000 86000000:06000000 86400000:06400000

86800000:06800000 86c00000:06c00000 87000000:07000000

87400000:07400000 87800000:07800000 87c00000:07c00000

a0000000:0153d000 c0000000:00d94000 c0400000:01615000

c0c00000:00041000 c1000000:00042000 c1400000:00043000

c1800000:00044000 c1c00000:00045000 c2000000:00046000

c2400000:00047000 c2800000:00048000 c2c00000:00049000

c3000000:0004a000 c3400000:0004b000 c3800000:0004c000

c3c00000:0004d000 c4000000:0004e000 c4400000:0000f000

c4800000:00050000 c4c00000:00051000 c5000000:00052000

c5400000:00053000 c5800000:00054000 c5c00000:00055000

c6000000:00056000 c6400000:00057000 c6800000:00058000

c6c00000:00059000 c7000000:0005a000 c7400000:0005b000

c7800000:0005c000 c7c00000:0005d000 c8000000:0005e000

c8400000:0005f000 c8800000:00020000 c8c00000:00021000

c9000000:00022000 c9400000:00023000 c9800000:00024000

c9c00000:00025000 ca000000:00026000 ca400000:00027000

ca800000:00028000 cac00000:00029000 cb000000:0002a000

cb400000:0002b000 cb800000:0002c000 cbc00000:0002d000

cc000000:0002e000 cc400000:0002f000 cc800000:002f0000

ccc00000:002f1000 cd000000:002f2000 cd400000:002f3000

cd800000:002f4000 cdc00000:002f5000 ce000000:002f6000

ce400000:00037000 ce800000:00038000 cec00000:00039000

cf000000:0003a000 cf400000:0003b000 cf800000:0003c000

cfc00000:0003d000 d0000000:0003e000 d0400000:0003f000

d0800000:00380000 d0c00000:00301000 d1000000:00302000

d1400000:00303000 d1800000:00304000 d1c00000:00305000

d2000000:00306000 d2400000:00307000 d2800000:00308000

d2c00000:00309000 d3000000:0030a000 d3400000:0030b000

d3800000:0030c000 d3c00000:0030d000 d4000000:0030e000

d4400000:0004f000 d4800000:00310000 d4c00000:00311000

e1000000:00315000 e1400000:010fe000 fc400000:0038d000

fc800000:0038e000 fcc00000:0038f000 fd000000:00390000

fd400000:00391000 fd800000:00392000 fdc00000:00393000

fe000000:00394000 fe400000:00395000 fe800000:00396000

fec00000:00397000 ff000000:00398000 ff400000:00399000

ff800000:0039a000 ffc00000:00031000

Memory Management

2007-11-17T07:50:00.000-08:00

Memory Management

Author: Prasad Dabak
Milind Borate
Sandeep Phadke
Published: October 1999
Copyright: 1999
Publisher: M&T Books

Abstract
This chapter examines memory models in Microsoft operating systems, examines how Windows NT uses features of the 80386 processor's architecture, and explores the function of virtual memory.
MEMORY MANAGEMENT HAS ALWAYS been one of the most important and interesting aspects of any operating system for serious developers. It is an aspect that kernel developers ignore. Memory management, in essence, provides a thumbnail impression of any operating system.

Microsoft has introduced major changes in the memory management of each new operating system they have produced. Microsoft had to make these changes because they developed all of their operating systems for Intel microprocessors, and Intel introduced major changes in memory management support with each new microprocessor they introduced. This chapter is a journey through the various Intel microprocessors and the memory management changes each one brought along with it in the operating system that used it.

MEMORY MODELS IN MICROSOFT OPERATING SYSTEMS

Early PCs based on Intel 8086/8088 microprocessors could access only 640K of RAM and used the segmented memory model. Consequently, good old DOS allows only 640K of RAM and restricts the programmer to the segmented memory model.

In the segmented model, the address space is divided into segments. Proponents of the segmented model claim that it matches the programmer’s view of memory. They claim that a programmer views memory as different segments containing code, data, stack, and heap. Intel 8086 supports very primitive segmentation. A segment, in the 8086 memory model, has a predefined base address. The length of each segment is also fixed and is equal to 64K. Some programs find a single segment insufficient. Hence, there are a number of memory models under DOS. For example, the tiny model that supports a single segment for code, data, and stack together, or the small model that allows two segments–one for code and the other for data plus stack, and so on. This example shows how the memory management provided by an operating system directly affects the programming environment.

The Intel 80286 (which followed the Intel 8086) could support more than 640K of RAM. Hence, programmers got new interface standards for accessing extended and expanded memory from DOS. Microsoft’s second-generation operating system, Windows 3.1, could run on 80286 in standard mode and used the segmented model of 80286. The 80286 provided better segmentation than the 8086. In 80286’s model, segments can have a programmable base address and size limit. Windows 3.1 had another mode of operation, the enhanced mode, which required the Intel 80386 processor. In the enhanced mode, Windows 3.1 used the paging mechanisms of 80386 to provide additional performance. The virtual 8086 mode was also used to implement multiple DOS boxes on which DOS programs could run.

Windows 3.1 does not make full use of the 80386’s capabilities. Windows 3.1 is a 16-bit operating system, meaning that 16-bit addresses are used to access the memory and the default data size is also 16 bits. To make full use of 80386’s capabilities, a 32-bit operating system is necessary. Microsoft came up with a 32-bit operating system, Windows NT. The rest of this chapter examines the details of Windows NT memory management. Microsoft also developed Windows 95 after Windows NT. Since both these operating systems run on 80386 and compatibles, their memory management schemes have a lot in common. However, you can best appreciate the differences between Windows NT and Windows 95/98 after we review Windows NT memory management. Therefore, we defer this discussion until a later section of this chapter.

WINDOWS NT MEMORY MANAGEMENT OVERVIEW

We’ll first cover the view Windows NT memory management presents to the outside world. In the next section, we explain the special features provided by Intel microprocessors to implement memory management. Finally, we discuss how Windows NT uses these features to implement the interface provided to the outside world.

Memory Management Interface—Programmer’s View
Windows NT offers programmers a 32-bit flat address space. The memory is not segmented; rather, it is 4GB of continuous address space. (Windows NT marked the end of segmented architecture–programmers clearly preferred flat models to segmented ones.) Possibly, with languages such as COBOL where you need to declare data and code separately, programmers view memory as segments. However, with new languages such as C and C++, data variables and code can be freely mixed and the segmented memory model is no longer attractive. Whatever the reason, Microsoft decided to do away with the segmented memory model with Windows NT. The programmer need not worry whether the code/data fits in 64K segments. With the segmented memory model becoming extinct, the programmer can breathe freely. At last, there is a single memory model, the 32-bit flat address space.

Windows NT is a protected operating system; that is, the behavior (or misbehavior) of one process should not affect another process. This requires that no two processes are able to see each other’s address space. Thus, Windows NT should provide each process with a separate address space. Out of this 4GB address space available to each process, Windows NT reserves the upper 2GB as kernel address space and the lower 2GB as user address space, which holds the user-mode code and data. The entire address space is not separate for each process. The kernel code and kernel data space (the upper 2GB) is common for all processes; that is, the kernel-mode address space is shared by all processes. The kernel-mode address space is protected from being accessed by user-mode code. The system DLLs (for example, KERNEL32.DLL, USER32.DLL, and so on) and other DLLs are mapped in user-mode space. It is inefficient to have a separate copy of a DLL for each process. Hence, all processes using the DLL or executable module share the DLL code and incidentally the executable module code. Such a shared code region is protected from being modified because a process modifying shared code can adversely affect other processes using the code.

Sharing of the kernel address space and the DLL code can be called implicit sharing. Sometimes two processes need to share data explicitly. Windows NT enables explicit sharing of address space through memory-mapped files. A developer can map a named file onto some address space, and further accesses to this memory area are transparently directed to the underlying file. If two or more processes want to share some data, they can map the same file in their respective address spaces. To simply share memory between processes, no file needs to be created on the hard disk.

BELOW THE OPERATING SYSTEM

In her book Inside Windows NT, Helen Custer discusses memory management in the context of the MIPS processor. Considering that a large number of the readers would be interested in a similar discussion that focuses on Intel processors, we discuss the topic in the context of the Intel 80386 processor (whose memory management architecture is mimicked by the later 80486 and Pentium series). If you are already conversant with the memory management features of the 80386 processor, you may skip this section entirely.

We now examine the 80386’s addressing capabilities and the fit that Windows NT memory management provides for it. Intel 80386 is a 32-bit processor; this implies that the address bus is 32-bit wide, and the default data size is as well. Hence, 4GB (232 bytes) of physical RAM can be addressed by the microprocessor. The microprocessor supports segmentation as well as paging. To access a memory location, you need to specify a 16-bit segment selector and a 32-bit offset within the segment. The segmentation scheme is more advanced than that in 8086. The 8086 segments start at a fixed location and are always 64K in size. With 80386, you can specify the starting location and the segment size separately for each segment.

Segments may overlap–that is, two segments can share address space. The necessary information (the starting offset, size, and so forth) is conveyed to the processor via segment tables. A segment selector is an index into the segment table. At any time, only two segment tables can be active: a Global Descriptor Table (GDT) and a Local Descriptor Table (GDT). A bit in the selector indicates whether the processor should refer to the LDT or the GDT. Two special registers, GDTR and LDTR, point to the GDT and the LDT, respectively. The instructions to load these registers are privileged, which means that only the operating system code can execute them.

A segment table is an array of segment descriptors. A segment descriptor specifies the starting address and the size of the segment. You can also specify some access permission bits with a segment descriptor. These bits specify whether a particular segment is read-only, read-write, executable, and so on. Each segment descriptor has 2 bits specifying its privilege level, called as the descriptor privilege level (DPL).

The processor compares the DPL with the Requested Privilege Level (RPL) before granting access to a segment. The RPL is dictated by 2 bits in the segment selector while specifying the address. The Current Privilege Level (CPL) also plays an important role here. The CPL is the DPL of the code selector being executed. The processor grants access to a particular segment only if the DPL of the segment is less than or equal to the RPL as well as the CPL. This serves as a protection mechanism for the operating system. The CPL of the processor can vary between 0 and 3 (because 2 bits are assigned for CPL). The operating system code generally runs at CPL=0, also called as ring 0, while the user processes run at ring 3. In addition, all the segments belonging to the operating system are allotted DPL=0. This arrangement ensures that the user mode cannot access the operating system memory segments.

It is very damaging to performance to consult the segment tables, which are stored in main memory, for every memory access. Caching the segment descriptor in special CPU registers, namely, CS (Code Selector), DS (Data Selector), SS (Stack Selector), and two general-purpose selectors called ES and FS, solves this problem. The first three selector registers in this list–that is, CS, DS, and SS–act as default registers for code access, data access, and stack access, respectively.

To access a memory location, you specify the segment and offset within that segment. The first step in address translation is to add the base address of the segment to the offset. This 32-bit address is the physical memory address if paging is not enabled. Otherwise this address is called as the logical or linear address and is converted to a physical RAM address using the page address translation mechanism (refer to Figure 4-1).

The memory management scheme is popularly known as paging because the memory is divided into fixed-size regions called pages. On Intel processors (80386 and higher), the size of one page is 4 kilobytes. The 32-bit address bus can access up to 4GB of RAM. Hence, there are one million (4GB/4K) pages.

Page address translation is a logical to physical address mapping. Some bits in the logical/linear address are used as an index in the page table, which provides a logical to physical mapping for pages. The page translation mechanism on Intel platforms has two levels, with a structure called page table directory at the second level. As the name suggests, a page table directory is an array of pointers to page tables. Some bits in the linear address are used as an index in the page table directory to get the appropriate page table to be used for address translation.

The page address translation mechanism in the 80386 requires two important data structures to be maintained by the operating system, namely, the page table directory and the page tables. A special register, CR3, points to the current page table directory. This register is also called Page Directory Base Register (PDBR). A page table directory is a 4096-byte page with 1024 entries of 4 bytes each. Each entry in the page table directory points to a page table. A page table is a 4096-byte page with 1024 entries of 4 bytes (32 bits) each. Each Page Table Entry (PTE) points to a physical page. Since there are 1 million pages to be addressed, out of the 32 bits in a PTE, 20 bits act as upper 20 bits of physical address. The remaining 12 bits are used to maintain attributes of the page.

Some of these attributes are access permissions. For example, you can denote a page as read-write or read-only. A page also has an associated security bit called as the supervisor bit, which specifies whether a page can be accessed from the user-mode code or only from the kernel-mode code. A page can be accessed only at ring 0 if this bit is set. Two other bits, namely, the accessed bit and the dirty bit, indicate the status of the page. The processor sets the accessed bit whenever the page is accessed. The processor sets the dirty bit whenever the page is written to. Some bits are available for operating system use. For example, Windows NT uses one such bit for implementing the copy-on-write protection. You can also mark a page as invalid and need not specify the physical page address. Accessing such a page generates a page fault exception. An exception is similar to a software interrupt. The operating system can install an exception handler and service the page faults. You’ll read more about this in the following sections.

32-bit memory addresses break down as follows. The upper 10 bits of the linear address are used as the page directory index, and a pointer to the corresponding page table is obtained. The next 10 bits from the linear address are used as an index in this page table to get the base address of the required physical page. The remaining 12 bits are used as offset within the page and are added to the page base address to get the physical address.

THE INSIDE LOOK

In this section, we examine how Windows NT has selectively utilized existing features of the 80386 processor’s architecture to achieve its goals.

Flat Address Space
First, let’s see how Windows NT provides 32-bit flat address space to the processes. As we know from the previous section, Intel 80386 offers segmentation as well as paging. So how does Windows NT provide a flat memory instead of a segmented one? Turn off segmentation? You cannot turn off segmentation on 80386. However, the 80386 processor enables the operating system to load the segment register once and then specify only 32-bit offsets for subsequent instructions. This is exactly what Windows NT does. Windows NT initializes all the segment registers to point to memory locations from 0 to 4GB, that is, the base is set as 0 and the limit is set as 4GB. The CS, SS, DS, and ES are initialized with separate segment descriptors all pointing to locations from 0 to 4GB. So now the applications can use only 32-bit offset, and hence see a 32-bit flat address space. A 32-bit application running under Windows NT is not supposed to change any of its segment registers.

Memory Management in C++

2007-11-17T07:43:00.000-08:00

Memory Management in C++

by Nathan C. Myers

Memory usage in C++ is as the sea come to land:
a tide rolls in, and sweeps out again,
leaving only puddles and stranded fish.
At intervals, a wave crashes ashore; but the
ripples never cease.

Introduction

Many programs have little need for memory management; they use a fixed amount of memory, or simply consume it until they exit. The best that can be done for such programs is to stay out of their way. Other programs, including most C++ programs, are much less deterministic, and their performance can be profoundly affected by the memory management policy they run under. Unfortunately, the memory management facilities provided by many system vendors have failed to keep pace with growth in program size and dynamic memory usage.

Because C++ code is naturally organized by class, a common response to this failure is to overload member operator new for individual classes. In addition to being tedious to implement and maintain, however, this piece-meal approach can actually hurt performance in large systems. For example, applied to a tree-node class, it forces nodes of each tree to share pages with nodes of other (probably unrelated) trees, rather than with related data. Furthermore, it tends to fragment memory by keeping large, mostly empty blocks dedicated to each class. The result can be a quick new/delete cycle that accidentally causes virtual memory thrashing. At best, the approach interferes with system-wide tuning efforts.

Thus, while detailed knowledge of the memory usage patterns of individual classes can be helpful, it is best applied by tuning memory usage for a whole program or major subsystem. The first half of this article describes an interface which can ease such tuning in C++ programs. Before tuning a particular program, however, it pays to improve performance for all programs, by improving the global memory manager. The second half of this article covers the design of a global memory manager that is as fast and space-efficient as per-class allocators.

But raw speed and efficiency are only a beginning. A memory management library written in C++ can be an organizational tool in its own right. Even as we confront the traditional problems involving large data structures, progress in operating systems is yielding different kinds of memory -- shared memory, memory-mapped files, persistent storage -- which must be managed as well. With a common interface to all types of memory, most classes need not know the difference. This makes quite a contrast with systems of classes hard-wired to use only regular memory.

Global Operator New

In C++, the only way to organize memory management on a larger scale than the class is by overloading the global operator new. To select a memory management policy requires adding a placement argument, in this case a reference to a class which implements the policy:

extern void* operator new(size_t, class Heap&);

When we overload the operator new in this way, we recognize that the regular operator new is implementing a policy of its own, and we would like to tune it as well. That is, it makes sense to offer the same choices for the regular operator new as for the placement version.

In fact, one cannot provide an interesting placement operator new without also replacing the regular operator new. The global operator delete can take no user parameters, so it must be able to tell what to do just by looking at the memory being freed. This means that the operator delete and all operators new must agree on a memory management architecture.

For example, if our global operators new were to be built on top of malloc(), we would need to store extra data in each block so that the global operator delete would know what to do with it. Adding a word of overhead for each object to malloc()'s own overhead (a total of 16 bytes, on most RISCs), would seem a crazy way to improve memory management. Fortunately, all this space overhead can be eliminated by bypassing malloc(), as will be seen later.

The need to replace the global operators new and delete when adding a placement operator new has profound effects on memory management system design. It means that it is impossible to integrate different memory management architectures. Therefore, the top-level memory management architecture must be totally general, so that it can support any policy we might want to apply. Total generality, in turn, requires absolute simplicity.

An Interface

How simple can we get? Let us consider some declarations. Heap is an abstract class:

class Heap {
protected:
virtual ~Heap();
public:
virtual void* allocate(size_t) = 0;
static Heap& whatHeap(void*);
};

(The static member function whatHeap(void*) is discussed later.) Heap's abstract interface is simple enough. Given a global Heap pointer, the regular global operator new can use it:

extern Heap* __global_heap;

inline void*
operator new(size_t sz)
{ return ::__global_heap->allocate(sz); }

Inline dispatching makes it fast. It's general too; we can use the Heap interface to implement the placement operator new, providing access to any private heap:

inline void*
operator new(size_t size, Heap& heap
{ return heap.allocate(size); }

What kind of implementations might we define for the Heap interface? Of course the first must be a general purpose memory allocator, class HeapAny. (HeapAny is the memory manager described in detail in the second half of this article.) The global heap pointer, used by the regular operator new defined above, is initialized to refer to an instance of class HeapAny:

extern class HeapAny __THE_global_heap;
Heap* __global_heap = &__THE_global_heap;

Users, too, can instantiate class HeapAny to make a private heap:

HeapAny& myheap = *new HeapAny;

and allocate storage from it, using the placement operator new:

MyType* mine = new(myheap) MyType;

As promised, deletion is the same as always:

delete mine;

Now we have the basis for a memory management architecture. It seems that all we need to do is provide an appropriate implementation of class Heap for any policy we might want. As usual, life is not so simple.

Complications

What happens if MyType's constructor itself needs to allocate memory? That memory should come from the same heap, too. We could pass a heap reference to the constructor:

mine = new(myheap) MyType(myheap);

and store it in the object for use later, if needed. However, in practice this approach leads to a massive proliferation of Heap& arguments -- in constructors, in functions that call constructors, everywhere! -- which penetrates from the top of the system (where the heaps are managed) to the bottom (where they are used). Ultimately, almost every function needs a Heap& argument. Applied earnestly, the result can be horrendous. Even at best, such an approach makes it difficult to integrate other libraries into a system.

One way to reduce the proliferation of Heap arguments is to provide a function to call to discover what heap an object is on. That is the purpose of the the Heap::whatHeap() static member function. For example, here's a MyType member function that allocates some buffer storage:

char* MyType::make_buffer()
{
Heap& aHeap = Heap::whatHeap(this);
return new(aHeap) char[BUFSIZ];
}

(If "this" points into the stack or static space, whatHeap() returns a reference to the default global heap.)

Another way to reduce Heap argument proliferation is to substitute a private heap to be used by the global operator new. Such a global resource calls for gingerly handling. Class HeapStackTop's constructor replaces the default heap with its argument, but retains the old default so it can be restored by the destructor:

class HeapStackTop {
Heap* old_;
public:
HeapStackTop(Heap& h);
~HeapStackTop();
};

We might use this as follows:

{ HeapStackTop top = myheap;
mine = new MyType;
}

Now space for the MyType object, and any secondary store allocated by its constructor, comes from myheap. At the closing bracket, the destructor ~HeapStackTop() restores the previous default global heap. If one of MyType's member functions might later want to allocate more space from the same heap, it can use whatHeap(); or the constructor can save a pointer to the current global heap before returning.

Creating a HeapStackTop object is very clean way to install any global memory management mechanism: a HeapStackTop object created in main() quietly slips a new memory allocator under the whole program.

Some classes must allocate storage from the top-level global heap regardless of the current default. Any object can force itself to be allocated there by defining a member operator new, and can control where its secondary storage comes from by the same techniques described above.

With HeapStackTop, many classes need not know about Heap at all; this can make a big difference when integrating libraries from various sources. On the other hand, the meaning of Heap::whatHeap() (or a Heap& member or argument) is easier to grasp; it is clearer, and therefore safer. While neither approach is wholly satisfactory, a careful mix of the two can reduce the proliferation of Heap& arguments to a reasonable level.

Uses for Private Heaps

But what can private heaps do for us? We have hinted that improved locality of reference leads to better performance in a virtual memory environment, and that a uniform interface helps when using special types of memory.

One obvious use for private heaps is as a sort of poor man's garbage collection:

Heap* myheap = new HeapTrash;
... // lots of calls to new(*myheap)
delete myheap;

Instead of deleting objects, we discard the whole data structure at one throw. The approach is sometimes called "lifetime management". Since the destructors are never called, you must carefully control what kind of objects are put in the heap; it would be hazardous ever to install such a heap as the default (with HeapStackTop) because many classes, including iostream, allocate space at unpredictable times. Dangling pointers to objects in the deleted heap must be prevented, which can be tricky if any objects secretly share storage among themselves. Objects whose destructors do more than just delete other objects require special handling; the heap may need to maintain a registry of objects that require "finalization".

But private heaps have many other uses that don't violate C++ language semantics. Perhaps the quietest one is simply to get better performance than your vendor's malloc() offers. In many large systems, member operator new is defined for many classes just so they may call the global operator new less often. When the global operator new is fast enough, such code can be deleted, yielding easier maintenance, often with a net gain in performance from better locality and reduced fragmentation.

An idea that strikes many people is that a private heap could be written that is optimized to work well with a particular algorithm. Because it need not field requests from the rest of the program, it can concentrate on the needs of that algorithm. The simplest example is a heap that allocates objects of only one size; as we will see later, however, the default heap can be made fast enough that this is no great advantage. A mark/release mechanism is optimal in some contexts (such as parsing), if it can be used for only part of the associated data structure.

When shared memory is used for interprocess communication, it is usually allocated by the operating system in blocks larger than the objects that you want to share. For this case a heap that manages a shared memory region can offer the same benefits that regular operator new does for private memory. If the interface is the same as for non-shared memory, objects may not need to know they are in shared memory. Similarly, if you are constrained to implement your system on an architecture with a tiny address space, you may need to swap memory segments in and out. If a private heap knows how to handle these segments, objects that don't even know about swapping can be allocated in them.

In general, whenever a chunk of memory is to be carved up and made into various objects, a Heap-like interface is called for. If that interface is the same for the whole system, then other objects need not know where the chunk came from. As a result, objects written without the particular use in mind may safely be instantiated in very peculiar places.

In a multi-threaded program, the global operator new must carefully exclude other threads while it operates on its data structures. The time spent just getting and releasing the lock can itself become a bottleneck in some systems. If each thread is given a private heap which maintains a cache of memory available without locking, the threads need not synchronize except when the cache becomes empty (or too full). Of course, the operator delete must be able to accept blocks allocated by any thread, but it need not synchronize if the block being freed came from the heap owned by the thread that is releasing it.

A heap that remembers details about how, or when, objects in it were created can be very useful when implementing an object- oriented database or remote procedure call mechanism. A heap that segregates small objects by type can allow them to simulate virtual function behavior without the overhead of a virtual function table pointer in each object. A heap that zero-fills blocks on allocation can simplify constructors.

Programs can be instrumented to collect statistics about memory usage (or leakage) by substituting a specialized heap at various places in a program. Use of private heaps allows much finer granularity than the traditional approach of shadowing malloc() at link time.

In the remainder of this article we will explore how to implement HeapAny efficiently, so that malloc(), the global operator new(size_t), the global operator new(size_t, Heap&), and Heap::whatHeap(void*) can be built on it.

A Memory Manager in C++

An optimal memory manager has minimal overhead: space used is but fractionally larger than the total requested, and the new/delete cycle time is small and constant. Many factors work against achieving this optimum.

In many vendor libraries, memory used by the memory manager itself, for bookkeeping, can double the total space used. Fragmentation, where blocks are free but unavailable, can also multiply the space used. Space matters, even today, because virtual memory page faults slow down your program (indeed, your entire computer), and swap space limits can be exceeded just as can real memory.

A memory manager can also waste time in many ways. On allocation, a block of the right size must be found or made. If made, the remainder of the split block must be placed where it can be found. On deallocation, the freed block may need to be coalesced with any neighboring blocks, and the result must be placed where it can be found again. System calls to obtain raw memory can take longer than any other single operation; a page fault that results when idle memory is touched is just a hidden system call. All these operations take time, time spent not computing results.

The effects of wasteful memory management can be hard to see. Time spent thrashing the swap file doesn't show up on profiler output, and is hard to attribute to the responsible code. Often the problem is easily visible only when memory usage exceeds available swap space. Make no mistake: poor memory management can multiply your program's running time, or so bog down a machine that little else can run.

Before buying (or making your customers buy) more memory, it makes sense to see what can be done with a little code.

Principles

A memory manager project is an opportunity to apply principles of good design: Separate the common case from special cases, and make the common case fast and cheap, and other cases tolerable; make the user of a feature bear the cost of its use; use hints; reuse good ideas. [Lampson]

Before delving into detailed design, we must be clear about our goals. We want a memory manager that satisfies the following:

# Speed:
It must be much faster than existing memory managers, especially for small objects. Performance should not suffer under common usage patterns, such as repeatedly allocating and freeing the same block.
# Low overhead:
The total size of headers and other wasted space must be a small percentage of total space used, even when all objects are tiny. Repeated allocation and deallocation of different sizes must not cause memory usage to grow without bound.
# Small working set:
The number of pages touched by the memory manager in satisfying a request must be minimal, to avoid paging delays in virtual memory systems. Unused memory must be returned to the operating system periodically.
# Robustness:
Erroneous programs must have difficulty corrupting the memory manager's data structures. Errors must be flagged as soon as possible, not allowed to accumulate. Out-of-memory events must be handled gracefully.
# Portability:
The memory manager must adapt easily to different machines.
# Convenience:
Users mustn't need to change code to use it.
# Flexibility:
It must be easily customized for unusual needs, without imposing any additional overhead.

Techniques

Optimal memory managers would be common if they were easily built. They are scarce, so you can expect that a variety of subtle techniques are needed even to approach the optimum.

One such technique is to treat different request sizes differently. In most programs, small blocks are requested overwhelmingly more often than large blocks, so both time and space overhead for them is felt disproportionately.

Another technique results from noting that there are only a few different sizes possible for very small blocks, so that each such size may be handled separately. We can even afford to keep a vector of free block lists for those few sizes.

A third is to avoid system call overhead by requesting memory from the operating system in big chunks, and by not touching unused (and possibly paged-out) blocks unnecessarily. This means data structures consulted to find a block to allocate should be stored compactly, apart from the unused blocks they describe.

The final, and most important, technique is to exploit address arithmetic which, while not strictly portable according to language standards, works well on all modern flat-memory architectures. A pointer value can be treated as an integer, and bitwise logical operations may be used on it to yield a new pointer value. In particular, the low bits may be masked off to yield a pointer to a header structure that describes the block pointed to. In this way a block need not contain a pointer to that information. Furthermore, many blocks can share the same header, amortizing its overhead across all. (This technique is familiar in the LISP community, where it is known as "page-tagging".)

With so many goals, principles, and techniques to keep track of, it should be no surprise that there are plenty of pitfalls to avoid. They will be discussed later.

A Design

The first major feature of the design is suggested by the final two techniques above. We request memory from the operating system in units of a large power of two (e.g. 64K bytes) in size, and place them so they are aligned on such a boundary. We call these units "segments". Any address within the segment may have its low bits masked off, yielding a pointer to the segment header. We can treat this header as an instance of the abstract class HeapSegment:

class HeapSegment {
public:
virtual void free(void*) = 0;
virtual void* realloc(void*) = 0;
virtual Heap& owned_by(void*) = 0;
};

The second major feature of the design takes advantage of the small number of small-block sizes possible. A segment (with a header of class HeapPageseg) is split up into pages, where each page contains blocks of only one size. A vector of free lists, with one element for each size, allows instant access to a free block of the right size. Deallocation is just as quick; no coalescing is needed. Each page has just one header to record the size of the blocks it contains, and the owning heap. The page header is found by address arithmetic, just like the segment header. In this way, space overhead is limited to a few percent, even for the smallest blocks, and the time to allocate and deallocate the page is amortized over all usage of the blocks in the page.

For larger blocks, there are too many sizes to give each a segment; but such blocks may be packed adjacent to one another within a segment, to be coalesced with neighboring free blocks when freed. (We will call such blocks "spans", with a segment header of type HeapSpanseg.) Fragmentation, the proliferation of free blocks too small to use, is the chief danger in span segments, and there are several ways to limit it. Because the common case, small blocks, is handled separately, we have some breathing room: spans may have a large granularity, and we can afford to spend more time managing them. A balanced tree of available sizes is fast enough that we can use several searches to avoid creating tiny unusable spans. The tree can be stored compactly, apart from the free spans, to avoid touching them until they are actually used. Finally, aggressive coalescing helps reclaim small blocks and keep large blocks available.

Blocks too big to fit in a segment are allocated as a contiguous sequence of segments; the header of the first segment in the sequence is of class HeapHugeseg. Memory wasted in the last segment is much less than might be feared; any pages not touched are not even assigned by the operating system, so the average waste for huge blocks is only half a virtual-memory page.

Dispatching for deallocation is simple and quick:

void operator delete(void* ptr)
{
long header = (long)ptr & MASK;
((HeapSegment*)header)->free(ptr);
}

HeapSegment::free() is a virtual function, so each segment type handles deallocation its own way. This allows different Heaps to coexist. If the freed pointer does not point to allocated memory, the program will most likely crash immediately. (This is a feature. Bugs that are allowed to accumulate are extremely difficult to track down.)

The classical C memory management functions, malloc(), calloc(), realloc(), and free() can be implemented on top of HeapAny just as was the global operator new. Only realloc() requires particular support.

The only remaining feature to implement is the function Heap::whatHeap(void* ptr). We cannot assume that ptr refers to heap storage; it may point into the stack, or static storage, or elsewhere. The solution is to keep a bitmap of allocated segments, one bit per segment. On most architectures this takes 2K words to cover the entire address space. If the pointer refers to a managed segment, HeapSegment::owned_by() reports the owning heap; if not, a reference to the default global heap may be returned instead. (In the LISP community, this technique is referred to as BBOP, or "big bag o' pages".)

Pitfalls

Where we depart from the principles of good design mentioned above, we must be careful to avoid the consequences. One example is when we allocate a page to hold a small block: we are investing the time to get that page on behalf of all the blocks that may be allocated in it. If the user frees the block immediately, and we free the page, then the user has paid to allocate and free a page just to use one block in it. In a loop, this could be much slower than expected. To avoid this kind of thrashing, we can add some hysteresis by keeping one empty page for a size if there are no other free blocks of that size. Similar heuristics may be used for other boundary cases.

Another pitfall results from a sad fact of life: programs have bugs. We can expect programs to try to free memory that was not allocated, or that has already been freed, and to clobber memory beyond the bounds of allocated blocks. The best a regular memory manager can do is to throw an exception as early as possible when it finds things amiss. Beyond that, it can try to keep its data structures out of harm's way, so that bugs will tend to clobber users' data and not the memory manager's. This makes debugging much easier.

Initialization, always a problem for libraries, is especially onerous for a portable memory manager. C++ offers no way to control the order in which libraries are initialized, but the memory manager must be available before anything else. The standard iostream library, with a similar problem, gets away by using some magic in its header file (at a sometimes intolerable cost in startup time) but we don't have even this option, because modules that use the global operator new are not obliged to include any header file. The fastest approach is to take advantage of any non-portable static initialization ordering available on the target architecture. (This is usually easy.) Failing that, we can check for initialization on each call to operator new or malloc(). A better portable solution depends on a standard for control of initialization order, which seems (alas!) unlikely to appear.

Measurements

Tests of vendor-supplied memory managers can yield surprising results. We know of a program that used 250 megabytes of swap space (before crashing when it ran out) when linked with the vendor-supplied library, but only a steady 50 megabytes after it was relinked with our memory manager. We have a simple presentation graphics animation program that uses less than half a megabyte of memory, but draws twice as fast when relinked.

Dividends

The benefits of careful design often go beyond the immediate goals. Indeed, unexpected results of this design include a global memory management interface which allows different memory managers to coexist. For most programs, though, the greatest benefit beyond better performance is that all the ad hoc apparatus intended to compensate for a poor memory manager may be ripped out. This leaves algorithms and data structures unobscured, and allows classes to be used in unanticipated ways.

Thanks to Paul McKenney and Jim Shur for their help in improving this article.

Tips for memory management!!

2007-11-10T18:31:00.000-08:00

1. For C/C++

a. Forget to delete an object.

Ex:

int* x;
x = new int();
....
x = y; (2) <<<< leak memory here

Because the memory referenced by x is now no longer referenced by x after we execute statement 2.

b.Create an array but only delete one object;
Solution: Use delete[] to delete an array instead of delete

c. Delete object twice:
int *x;
int *y;

x = new int();
y = x;

...Do sth...
delete x;
...Do sth...
delete y;

--> crash here.

d. Create an object in dll but delete it outside dll. The correct way is call the function which delete object in dll. Dll should provide the way to delete object in its process.

e. Always check the array boundary before access it or you will be crashed and program will have the buffer overflow.

Visitor Pattern--How to implement

2007-10-13T00:38:00.000-07:00

Today, i want to introduce about Visitor Pattern.

Goal: Why do we have to use Visitor Pattern? The answer is we need to separate the algorithm and the object structure. To see it clearly, we will discuss about it.
The Visitor pattern is also the classic technique for recovering lost type information without resorting to dynamic casts.

In this class diagram, a visitor can be considered as a different algorithm.
A Object Structure is an object which the algorithm work on it. This ObjectStructure has a complex structure and we want to separate the structure from algorithm. To do it, we have to give the algorithm the part of an ObjectStructure. This part of object will do its own task by using the algorithm which give to it.

Example:

interface Visitor {

   void visit(Wheel wheel);
void visit(Engine engine);
void visit(Body body);
void visit(Car car);
}


interface Visitable {
void accept(Visitor visitor);
}

class Wheel implements Visitable {
private String name;
Wheel(String name) {
    this.name = name;
}
String getName() {
    return this.name;
}
public void accept(Visitor visitor) {
    visitor.visit(this);
}
}

class Engine implements Visitable{
public void accept(Visitor visitor) {
    visitor.visit(this);
}
}

class Body implements Visitable{
public void accept(Visitor visitor) {
    visitor.visit(this);
}
}

class Car implements Visitable {
private Engine  engine = new Engine();
private Body    body   = new Body();
private Wheel[] wheels
    = { new Wheel("front left"), new Wheel("front right"),
        new Wheel("back left") , new Wheel("back right")  };
public Engine getEngine() {
    return this.engine;
}
public Body getBody() {
    return this.body;
}
public Wheel[] getWheels() {
    return this.wheels;
}
public void accept(Visitor visitor) {
    visitor.visit(this);
    engine.accept(visitor);
    body.accept(visitor);
    for(int i = 0; i < class="me1">length; i++) {
        Wheel wheel = wheels[i];
        wheel.accept(visitor);
    }
}
}

class PrintVisitor implements Visitor {

public void visit(Wheel wheel) {
    System.out.println("Visiting "+ wheel.getName()
                        + " wheel");
}
public void visit(Engine engine) {
    System.out.println("Visiting engine");
}
public void visit(Body body) {
    System.out.println("Visiting body");
}
public void visit(Car car) {
    System.out.println("Visiting car");
}

}

class DoVisitor implements Visitor {
public void visit(Wheel wheel) {
    System.out.println("Steering my wheel");
}
public void visit(Engine engine) {
    System.out.println("Starting my engine");
}
public void visit(Body body) {
    System.out.println("Moving my body");
}
public void visit(Car car) {
    System.out.println("Vroom!");
}
}

public class VisitorDemo {
static public void main(String[] args){
    Car car = new Car();
    Visitor visitor = new PrintVisitor();
    Visitor doVisitor = new DoVisitor();
    car.accept(visitor);
    car.accept(doVisitor);
}
}

As we can see, a car has some part: Wheel, Engine, Body, Car.
The first algorithm: PrintVisitor( Print) and the second algorithm has different
behavior to four part of car and do not care about the structure of Car ( refer to
the main function). It is very important when implement the complex algorithm.
It is also useful when maintaining class. We can easily add new algorithm without
affecting the others algorithm and object structure.

PHP learning--Part 5

2007-09-30T00:52:00.000-07:00

Variables

+ Variable names follow the same rules as other labels in PHP. A valid variable name starts with a letter or underscore, followed by any number of letters, numbers, or underscores. As a regular expression, it would be expressed thus: '[a-zA-Z_\x7f-\xff][a-zA-Z0-9_\x7f-\xff]*'

Note: For our purposes here, a letter is a-z, A-Z, and the ASCII characters from 127 through 255 (0x7f-0xff).

Varible assignment
+ Assigned by value: That is to say, when you assign an expression to a variable, the entire value of the original expression is copied into the destination variable. This means, for instance, that after assigning one variable's value to another, changing one of those variables will have no effect on the other
+ Assigned by reference: This means that the new variable simply references (in other words, "becomes an alias for" or "points to") the original variable.Changes to the new variable affect the original, and vice versa.

To assign by reference, simply prepend an ampersand (&) to the beginning of the variable which is being assigned (the source variable)
Example:

$bar = &$foo;

Note:

PHP.Net: Relying on the default value of an uninitialized variable is problematic in the case of including one file into another which uses the same variable name. It is also a major security risk with register_globals turned on. E_NOTICE level error is issued in case of working with uninitialized variables, however not in the case of appending elements to the uninitialized array. isset() language construct can be used to detect if a variable has been already initialized.

Comment: The good behaviour is always initialize a varible before using it. It is always recommended on every programming language not only for PHP.

Variable scope

The scope of a variable is the context within which it is defined. For the most part all PHP variables only have a single scope. This single scope spans included and required files as well.

Example:

 
<?php
 $a = 1;
include 'b.inc';
?>

Varible $a will be available within the included b.inc script.

+ Any variable used inside a function is by default limited to the local function scope. For example:

 <?php
$a = 1; /* global scope */ 

function Test()
{ 
    echo $a; /* reference to local scope variable */ 
} 

Test();
?>

This script will not produce any output because the echo statement refers to a local version of the $a variable, and it has not been assigned a value within this scope

To used a global variable in the function we used the global keyword

Example:

 <?php
$a = 1;
$b = 2;

function Sum()
{
    global $a, $b;

    $b = $a + $b;
} 

Sum();
echo $b;
?>

Variable a, b in function is declared as global variable, all the references to either variable will refer to the global function.

The second way to access varibles from global scope is to use the special PHP-defined $GLOBALS Array. This $GLOBALS variable is the superglobal and can be used in any scope.

Static variable:
The static varible is available in local function scope but it is does notlose its value when program excution leaves this scope. Consider the following example:

 
<?php
function Test()
{
    static $a = 0; //just for defined static variable.
    echo $a;
    $a++;
}
?>

When we call Test the first times, it output: 0.
The second times, it output 1;

References with global and static variables

This can lead to unexpected behaviour which the following example addresses. So do not use it.

Next part will be in next day.
I want thank to php.net with the manual. This part what i learn from their manual.

PHP learning--part 4

2007-09-29T09:28:00.000-07:00

PHP Types:

About the primitive types, PHP has 4 primitive type:
+ boolean
+ integer
+ float
+ string

Two compound types:
+ array
An array in PHP is actually an ordered map. A map is a type that maps values to keys. This type is optimized in several ways, so you can use it as a real array, or a list (vector), hashtable (which is an implementation of a map), dictionary, collection, stack, queue and probably more. Because you can have another PHP array as a value, you can also quite easily simulate trees.
Note: It is so powerful, but complex, not easy to understand.

Example1: 1-dimension array ( or map)

 <?php
$arr = array("foo" => "bar", 12 => true);
echo $arr["foo"]; // bar
echo $arr[12];    // 1
?>

Example 2: 2 dimension array

 
<?php
$arr = array("somearray" => array(6 => 5, 13 => 9, "a" => 42));
echo $arr["somearray"][6];    // 5
echo $arr["somearray"][13];   // 9
echo $arr["somearray"]["a"];  // 42
?>

Array do's and don'ts

Why is $foo[bar] wrong?

If there are no constant define bar, PHP will automatically change bar into 'bar' and use that.
You will mismatch value define.


error_reporting(E_ALL);
ini_set('display_errors', true);
ini_set('html_errors', false);
// Simple array:
$array = array(1, 2);
$count = count($array);
for ($i = 0; $i < $count; $i++) {
   echo "\nChecking $i: \n";
   echo "Bad: " . $array['$i'] . "\n";
   echo "Good: " . $array[$i] . "\n";
   echo "Bad: {$array['$i']}\n";
   echo "Good: {$array[$i]}\n";
}
?>

The above example will output:

Checking 0:  Notice: Undefined index:  $i in /path/to/script.html on line 9 Bad:  Good: 1 Notice: Undefined index:  $i in /path/to/script.html on line 11 Bad:  Good: 1  Checking 1:  Notice: Undefined index:  $i in /path/to/script.html on line 9 Bad:  Good: 2 Notice: Undefined index:  $i in /path/to/script.html on line 11 Bad:  Good: 2 So array is flexible, but using it carefully.

+ Object

 
Object Initialization
<?php
class foo
{
   function do_foo()
   {
       echo "Doing foo.";
   }
}

$bar = new foo;
$bar->do_foo();
?>

Converting to object:

Rule: If an object is converted to an object, it is not modified. If a value of any other type is converted to an object, a new instance of the stdClass built in class is created. If the value was NULL, the new instance will be empty. Array converts to an object with properties named by array keys and with corresponding values. For any other value, a member variable named scalar will contain the value.

Example:

Convert string to object.

 <?php
$obj = (object) 'ciao';
echo $obj->scalar;  // outputs 'ciao'
?>

Resource

A resource is a special variable, holding a reference to an external resource. Resources are created and used by special functions

Converting to resource

As resource types hold special handlers to opened files, database connections, image canvas areas and the like, you cannot convert any value to a resource.

NULL

The special NULL value represents that a variable has no value

The type of a variable is usually not set by the programmer; rather, it is decided at runtime by PHP depending on the context in which that variable is used.-->Varible type in PHP equal type variant in many other programming language.

That's all. We will go to detail in next day.

PHP learning--part 3

2007-09-23T09:45:00.001-07:00

PHP syntax:

PHP code have to be put in
<?php code ?>
Example:
<?php echo 'While this is going to be parsed.'; ?>

There are 2 layer when parse a php file. First the PHP file compile with PHP parser. The output is HTML. Then the HTML output will be parse by HTML parser and then output into browser client. It is complex process behind.

When PHP parses a file, it looks for opening and closing tags, which tell PHP to start and stop interpreting the code between them. Parsing in this manner allows php to be embedded in all sorts of different documents, as everything outside of a pair of opening and closing tags is ignored by the PHP parser. Most of the time you will see php embedded in HTML documents, as in this example.

 <?php
if ($expression) { 
    ?>
    <strong>This is true.</strong>
    <?php 
} else { 
    ?>
    <strong>This is false.</strong>
    <?php 
}
?>

How this work?

I will explain:
First, when PHP parser meet first open tag <?php, it will start compile php code until it meet
close tag ?>. So the statment if ($expression) { will be put into parser. But the if statement is not completed. So it will wait until the if structure complete and output. Because the
$expression is default false, so the block: <strong>This is true.</strong> will be ignore. The compiler is enough intelligent to do this. When it meet else part, it will ouput <strong>This is false.</strong> to html parser. Then html parser will do its job: output to client browser.
Hope it is clear.

There are four different pairs of opening and closing tags which can be used in php. Two of those, <?php ?> and <script language="php"> </script>, are always available. The other two are short tags and ASP style tags, and can be turned on and off from the php.ini configuration file. As such, while some people find short tags and ASP style tags convenient, they are less portable, and generally not recommended.

Note: Also note that if you are embedding PHP within XML or XHTML you will need to use the <?php ?> tags to remain compliant with standards.

All types of php opening and closing tags:

 1. <?php echo 'if you want to serve XHTML or XML documents, do like this'; ?>

2.  <script language="php">
        echo 'some editors (like FrontPage) don\'t
              like processing instructions';
    </script>

3.  <? echo 'this is the simplest, an SGML processing instruction'; ?>
    <?= expression ?> This is a shortcut for "<? echo expression ?>"

4.  <% echo 'You may optionally use ASP-style tags'; %>
    <%= $variable; # This is a shortcut for "<% echo . . ." %>

PHP learning -- Part 2

2007-09-23T01:40:00.000-07:00

The second things i want to know: which do we need to develop a PHP program?

1. Web server: Apache(recommend) or IIS(not recommend)
2. DBMS such as Mysql(recommend) or others
3. PHP engine (Of course). Download at php.net

But i recommend on WAMP. It wrapped all module in one. It is easier for you to develop.

After install WAMP at port 80, i want to run the first hello world to have a look at PHP.

Although now we do not understand what its mean, it will give us an overview.

First hello world program written in PHP.

File: hello.php


<html>
 <head>
  <title>PHP Test</title>
 </head>
 <body>
 <?php echo '<p>Hello World</p>'; ?>
 </body>
</html>

To make in run, we start wamp server. Put this file into sub folder www in wamp folder.
Open browser, and enter URL: localhost/hello.php

The example will appear:
Hello World

Cool. We did a very simple program. Now we start look inside PHP in next post

PHP Learning--Part 1

2007-09-23T01:24:00.000-07:00

Today, i learn PHP language. I know a lot about C/C++, Delphi, Java but nothing about web programming(such as PHP). So now i will self-study PHP. I know it is difficult when you are starting learning new language. But when you have base knowledge. It will be easier.
First glance at PHP:

PHP, which stands for "PHP: Hypertext Preprocessor" is a widely-used Open Source general-purpose scripting language that is especially suited for Web development and can be embedded into HTML. Its syntax draws upon C, Java, and Perl, and is easy to learn. The main goal of the language is to allow web developers to write dynamically generated web pages quickly, but you can do much more with PHP.

That is definition in PHP.net. But in my opinion, in short way, PHP just a server side script run in server side. That's all.

Now i want to know the key feature of PHP. Let's start by study:
1. Server-side scripting:

This is the most traditional and main target field for PHP. You need three things to make this work. The PHP parser (CGI or server module), a web server and a web browser. You need to run the web server, with a connected PHP installation. You can access the PHP program output with a web browser, viewing the PHP page through the server. All these can run on your home machine if you are just experimenting with PHP programming

2.Command line scripting:

You can make a PHP script to run it without any server or browser. You only need the PHP parser to use it this way. This type of usage is ideal for scripts regularly executed using cron (on *nix or Linux) or Task Scheduler (on Windows). These scripts can also be used for simple text processing tasks
Note: It is like batch processing or Shell Programming. Good for system task such as make server log, backup....

3.Writing desktop applications. PHP is probably not the very best language to create a desktop application with a graphical user interface, but if you know PHP very well, and would like to use some advanced PHP features in your client-side applications you can also use PHP-GTK to write such programs. You also have the ability to write cross-platform applications this way. PHP-GTK is an extension to PHP, not available in the main distribution. If you are interested in PHP-GTK, visit http://gtk.php.net/.
Note: So cool. I do not know about this. Maybe i tried. ^_^ I only know that PHP is for Web development.

4. Database support:
Adabas D InterBase PostgreSQL
dBase FrontBase SQLite
Empress mSQL Solid
FilePro (read-only) Direct MS-SQL Sybase
Hyperwave MySQL Velocis
IBM DB2 ODBC Unix dbm
Informix Oracle (OCI7 and OCI8)
Ingres Ovrimos

We only need some key database server like MySQL, Oracle, Innobase, DB2, SQL Server(not see in the list. I will find the way to work around.).

The taming of the thread

2007-09-21T19:58:00.000-07:00

A process-driven approach to avoid thread death

Threads can be nasty beasts. This is partly attributed to their delicate nature. Threads can die. If the causes of thread death are not in your code, then MutableThread may keep your code running. This article explores a solution using an object-oriented, problem-solution method.

The problem

The first question to ask in the problem-solution method is, of course: What is the problem? The problem statement is easy to generate in existing systems since the problem generally causes the trouble. In this case, the problem is: The threads die, and the application stops running. No exception is thrown since the cause of thread death is external to the application.
The what and the how

It is extremely important to isolate the objective from the technical aspects of a possible solution. If design concerns influence the requirements early on, creative solutions might be prevented. It is a designer's nature to think about how to do everything and therefore his temptation to avoid challenges that might result in a superior solution by declaring them problematic, too ambitious, or even impossible.

The it-would-be-wonderful-if statement is a liberating way to imagine an ideal world. It produces the ideal what without regard to design limitations.

Let's define our what: Objects do not depend upon particular threads. If a thread should fail, the thread is recreated, and the object continues to run. This should be as unobtrusive as possible.
Scenarios can clarify the requirement and save time

Following the what review exercise, run through some scenarios the particular what will do. This first test can be done without spending a dime on design.

To generate some scenarios, simply imagine the potential objects created by the what and consider the possibilities that could happen. Try to think outside the box.

So, let's say we have an object that runs in a thread. We know the thread might die, and we need to deal with the consequences.

We must provide memory management to prevent so-called memory leaks. However, memory management has nothing directly to do with our thread-oriented requirement—it's beyond the scope of the what.

The crucial issue is that we want to maintain thread operation. There are a few possible subscenarios:

1. Sunny day: All is well. Just restart the thread and keep running.
2. Rainy day: Bad things happen. The object is somehow corrupted.
3. Typhoon: Really bad things happen. The JVM may be unstable.

Thinking more carefully, Scenario 2 describing corrupted data and Scenario 3 describing an unstable JVM might not affect this application. For this application, failure is the worst possibility. For others, it may be necessary to validate data or even JVM integrity. But for our simple case, the sunny day scenario suffices. Therefore, this design will not include data checking or object recreation in a new JVM. In a more robust design, the object persistence might be refreshed to validate data. In a worst-case scenario, the entire JVM should shutdown and restart.

The how: Identify objects and create a design

We must identify the objects involved in the design. First, there should be a detection object; we can call this object the watchdog. In our particular design, the watchdog watches all other threads. The watchdog runs in its own thread and has a collection of references to other threads so it can monitor them and make sure they're all alive. Conveniently, the Thread class provides an isAlive() method to determine if a thread is alive. The watchdog uses this method to detect each thread in its collection. If a thread fails, it's the watchdog's responsibility to report it.

For more robustness, this design will include a second "dog," the beta dog (the watchdog is the alpha dog). The beta dog's purpose is simply to check that the alpha dog is alive. The alpha dog also detects the beta dog.
Watchdogs

The ThreadWatchDog is a particular MutableThread instance that monitors threads, and monitored threads must be either MutableThread or Thread (or their descendants). The watchdog runs through the collection of threads and invokes the isAlive() method. When it notices that a thread is dead, it uses reCreate() to recreate the thread if it is a mutable thread. Otherwise, it simply reports the failure.

Here's how this looks (in the ThreadWatchDog test program associated with this article):


      if (lTestThread instanceof MutableThread)
            {
                if (!((MutableThread)lTestThread).isAlive())

This tests the mutable thread to see if it is alive. In the case of a MutableThread, it reports an exception and then attempts to restart the thread, if possible, with the following code:


ReportingExceptionHandler.processException( new ReportingException("Mutable Thread " + lThreadKey + " is dead"));
     try {
         // Attempt to restart the thread by clearing and restarting
         ((MutableThread)lTestThread).createThread();
         ((MutableThread)lTestThread).start();

For a MutableThread, the ThreadKey is the name assigned to the thread when it is created. The application sets up the threads and assigns them to the watchdog on startup. This is done as follows:


        TestThread threadOne = new TestThread();
        threadOne.setName("threadOne");
        TestThread threadTwo = new TestThread();
        threadTwo.setName("threadTwo");
        TestMutableThread threadMutable = new TestMutableThread();
        threadMutable.setName("threadMutable");
        ThreadWatchDog.getInstance().put(threadOne);
        ThreadWatchDog.getInstance().put(threadTwo);
        ThreadWatchDog.getInstance().put(threadMutable);
        threadOne.start();
        System.out.println("TEST: Thread One started");
        threadTwo.start();
        System.out.println("TEST: Thread Two started");
        threadMutable.start();
        MutableThread lWatchDog = ThreadWatchDog.getInstance();
        System.out.println("TEST: Starting the watchdog");
        lWatchDog.start();

This starts up the watchdog(s), and thread monitoring is now active. Note clearly that the threads should be started up before initializing the watchdogs, or things will get really confusing.

Note that the put() method adds threads to the ThreadWatchDog. This adds the thread to the collection. The put() method is also overloaded with a put(MutableThread mutableThread). This is because MutableThread isn't really a Thread; rather it implements the MutableRunnable interface, much as Thread implements the Runnable interface.

The MutableThread includes a handle to the actual Thread, and this can be recreated, which replaces the thread owned by the mutable thread:


   public void createThread() {
        mThisThread = new Thread(this, mThreadName);
        mThisThread.setPriority(mPriority);
    }

Note that the mThisThread is created by passing the this through the thread constructor. That allows the current object to be assigned to the new thread.

The actual thread is encapsulated within the MutableThread and can be recreated and restarted.

The deprecated Thread.stop() method is used in the test program to show what happens when threads die:


      System.out.println("TEST: Stopping threadOne");
        threadOne.stop();
        System.out.println("TEST: Stopping threadMutable");
        threadMutable.stop();

Later in the test program, we even kill the alpha watchdog to make sure the failure is detected and reported. The watchdogs are named internally and do not need to be named as do the application threads.
More on MutableThread

The MutableThread class does not use thread groups, but this function can be added. The thread owned by the MutableThread is not accessible by any other class because the thread should not be referenced anywhere else. Users of this class should not be affected if the thread is replaced. The class is abstract because it does not implement the run() method. This must be implemented by any class that wants to be an instance of MutableThread.

Note that the MutableThread implements the Runnable interface, and the internal thread created with the new Thread(Runnable, String) then invokes the run() method within the thread's implementing Runnable class. The string passed is the name. All necessary attributes are retained within the MutableThread object so that the thread name and priority are assigned to a new thread when the createThread() method is invoked.
Explore mutation

That's about it. Download the source code file that accompanies this article and extract various bits to explore the notion of mutable threads and watchdogs on your own.

mutablethread.jar contains all the code for MutableThread and the watchdogs. I've included a PC batch file, run.bat, to help you invoke the test program. Simply type run in the same directory as the mutablethread.jar and run.bat files. The code includes a logger class that logs to the console, but this can be easily modified to a logging system such as log4j. There is also a ReportingException that handles nesting and reports the exception that a thread has died.

Finally, don't forget to summarize requirements and separate the requirements from design concerns. Following this analysis and design approach can be a big advantage in creating more reliable and extensible systems.

Detect and remove hardware problem with BUGDOCTOR.COM

2007-09-16T05:26:00.000-07:00

BUGDOCTOR.COM the antidote to all your computer error woes. New Bug-Detecting and Removal Software Bringing Back Your Computer's Pulse

March 9, 2005 - Are you one of the billions of computer user’s whose PC is infected with a pesky infection? SOFTWARE DOCTOR INC. today announced the worldwide public release of BugDoctor, the most advanced error-removal software on the market. BugDoctor is a new computer tool that seamlessly scans for, detects and deletes a comprehensive range of hard drive bugs. It roots out potentially crash-inducing errors at their source, ferreting deep into PC application sections-such as Active X Com, device drivers, corrupt help files, and Windows startup applications and shortcuts-where these parasites hide. The program is available for complimentary download at BugDoctor.com and an initial error scan is available free of charge.

According to a National Cyber Security study conducted in June 2003, 9 out of 10 PCs connected to the Internet contain spyware. Even more startling, 94% of PCs are infected with spyware-associated bugs and errors that if not repaired properly can put them at extreme risk for system failure. While many adware and spyware removal programs are effective in eradicating the annoying popups that such malicious programs carry with them, they leave behind registry keys that not only slow down system performance but also pose the potential for an outright system crash.

If you are noticing such computer problems as slow startups, troublesome shut-downs, unresponsive programs, or runtime, 404 or msg errors, then your computer could be at high risk for permanent failure. Before tossing thousands of money into a new system, however, you owe it to yourself to give BugDoctor a try. Chris V., a satisfied client, is glad he did: "I was going to invest over $2,500 in a new computer, but I decided to try BugDoctor first and I’m glad I did. Now my PC runs like new again, saving me hundreds of dollars and tons of time."

Not only will BugDoctor repair every type of aforementioned computer error, but it will also boost your system’s speed and performance by as much as 300%. The program is easy and safe to use and regular scanning and deletion will result in a more stable and faster running Windows environment.

To download the new BugDoctor error-removal software and to scan your own PC before it’s too late, visit the company online at www.BugDoctor.com. For more information on how BugDoctor can protect and save your computer from debilitating system bugs, contact Mike Delrue at support@bugdoctor.com.

###
This press release has been posted by Software Submit.NET, website and software submission on-line service, avaliable worldwide. For details contact Software Submit.NET at http://www.softwaresubmit.net

About This Release
If you have any questions regarding information in these press releases please contact the organization listed in the press release. Issuers of press releases and not PR Leap are solely responsible for the accuracy of the content.

Builder pattern

2007-09-15T17:43:00.000-07:00

Today i want to introduce Builder Pattern.

1. First we need to know the Builder Pattern class diagram:
Class diagram:

Now we will explain the meaning of those class:

Builder: An interface used to create complex object.

Concrete Builder: An implementation of Builder. Builder give us the way to construct object. And Concrete Builder show us how to do in detail.

Director: The Director class is responsible for managing the correct sequence of object creation. It receives a Concrete Builder as a parameter and executes the necessary operations on it.

Explanation:
We have a complex object with many part. To create object, we have to create many sub item in it. For example: We have a pizza. There are many type of pizza. Each type has its own way to do pizza.
For Hawaiian pizza: it make from cross, mild, ham+pineapple
For Spicy pizza: it make from pan baked, hot, pepperoni+salami

So we design as below:
+ Builder define how to create pizza. It provide the mean to create pizza:
public void setDough(String dough)
public void setSauce(String sauce)
public void setTopping(String topping)

For each kind of pizza( a concrete builder) , we will give pizza a suitable thing


/** "Product" */
class Pizza
{
    private String dough = "";
    private String sauce = "";
    private String topping = "";
 
    public void setDough(String dough)
    { this.dough = dough; }
    public void setSauce(String sauce)
    { this.sauce = sauce; }
    public void setTopping(String topping)
    { this.topping = topping; }
}

/** "Abstract Builder" */
abstract class PizzaBuilder
{
    protected Pizza pizza;
 
    public Pizza getPizza()
    { 
        return pizza; 
    }
    public void createNewPizzaProduct()
    { 
        pizza = new Pizza(); 
    }
 
    public abstract void buildDough();
    public abstract void buildSauce();
    public abstract void buildTopping();
}
 
/** "ConcreteBuilder" */
class HawaiianPizzaBuilder extends PizzaBuilder
{
    public void buildDough()
    { 
        pizza.setDough("cross"); 
    }
    public void buildSauce()
    { 
        pizza.setSauce("mild"); 
    }
    public void buildTopping()
    { 
        pizza.setTopping("ham+pineapple"); 
    }
}
 
/** "ConcreteBuilder" */
class SpicyPizzaBuilder extends PizzaBuilder
{
    public void buildDough()
    { 
        pizza.setDough("pan baked"); 
    }
    public void buildSauce()
    { 
        pizza.setSauce("hot"); 
    }
    public void buildTopping()
    { 
        pizza.setTopping("pepperoni+salami"); 
    }
}

But we need to describe the sequence to do pizza(step by step)
so we need to implement Director and give it Builder as parameter


class Waiter
{
    private PizzaBuilder pizzaBuilder;
 
    public void setPizzaBuilder(PizzaBuilder pb)
    {
        pizzaBuilder = pb; 
    }
    public Pizza getPizza()
    { 
        return pizzaBuilder.getPizza(); 
    }
 
    public void constructPizza()
    {
        pizzaBuilder.createNewPizzaProduct();
        pizzaBuilder.buildDough();
        pizzaBuilder.buildSauce();
        pizzaBuilder.buildTopping();
    }
}

Finally: We use it:


class BuilderExample
{
    public static void main(String[] args)
    {
        Waiter waiter = new Waiter();
        PizzaBuilder hawaiianPizzaBuilder = new HawaiianPizzaBuilder();
        PizzaBuilder spicyPizzaBuilder = new SpicyPizzaBuilder();
 
        waiter.setPizzaBuilder( hawaiianPizzaBuilder );
        waiter.constructPizza();
 
        Pizza pizza = waiter.getPizza();
    }
}

Hope it can help you to understand builder pattern

EJB best practices

2007-09-07T10:00:00.000-07:00

EJB best practices: Build a better exception-handling framework

Deliver more useful exceptions without sacrificing clean code

Level: Intermediate

Brett McLaughlin (brett@newInstance.com), Author and Editor, O'Reilly Media Inc.

01 Jan 2003

Enterprise applications are often built with little attention given to exception handling, which can result in over-reliance on low-level exceptions such as java.rmi.RemoteException and javax.naming.NamingException. In this installment of EJB Best Practices, Brett McLaughlin explains why a little attention goes a long way when it comes to exception handling, and shows you two simple techniques that will set you on the path to building more robust and useful exception handling frameworks.

In previous tips in this series, exception handling has been peripheral to our core discussion. One thing you may have picked up, however, is that we've consistently distanced low-level exceptions from the Web tier. Rather than have the Web tier handle exceptions such as java.rmi.RemoteException or javax.naming.NamingException, we've supplied exceptions like ApplicationException and InvalidDataException to the client.

Remote and naming exceptions are system-level exceptions, whereas application and invalid-data exceptions are business-level exceptions, because they deliver more applicable business information. When determining what type of exception to throw, you should always first consider the tier that will handle the reported exception. The Web tier is generally driven by end users performing business tasks, so it's better equipped to handle business-level exceptions. In the EJB layer, however, you're performing system-level tasks such as working with JNDI or databases. While these tasks will eventually be incorporated into business logic, they're best represented by system-level exceptions like RemoteException.

Theoretically, you could have all of your Web tier methods expect and respond to a single application exception, as we did in some of our previous examples. But that approach wouldn't hold up over the long run. A far better exception-handling scheme would be to have your delegate methods throw more specific exceptions, which are ultimately more useful to the receiving client. In this tip, we'll discuss two techniques that will help you create more informative, less generalized exceptions, without generating a lot of unnecessary code.

Nested exceptions

The first thing to think about when designing a solid exception-handling scheme is the abstraction of what I call low-level or system-level exceptions. These are generally core Java exceptions that report errors in network traffic, problems with JNDI or RMI, or other technical problems in an application. RemoteException, EJBException, and NamingException are common examples of low-level exceptions in enterprise Java programming.

These exceptions are fairly meaningless, and can be especially confusing when received by a client in the Web tier. A client that invokes purchase() and receives back a NamingException has little to work with when it comes to resolving the exception. At the same time, your application code may need to access the information within these exceptions, so you can't simply throw out or ignore them.

The answer is to provide a more useful type of exception that also contains a lower-level exception. Listing 1 shows a simple ApplicationException that does just this:

Listing 1. A nested exception

package com.ibm;

import java.io.PrintStream;
import java.io.PrintWriter;

public class ApplicationException extends Exception {

     /** A wrapped Throwable */
     protected Throwable cause;

     public ApplicationException() {
         super("Error occurred in application.");
     }

     public ApplicationException(String message)  {
         super(message);
     }

     public ApplicationException(String message, Throwable cause)  {
         super(message);
         this.cause = cause;
     }

     // Created to match the JDK 1.4 Throwable method.
     public Throwable initCause(Throwable cause)  {
         this.cause = cause;
         return cause;
     }

     public String getMessage() {
         // Get this exception's message.
         String msg = super.getMessage();

         Throwable parent = this;
         Throwable child;

         // Look for nested exceptions.
         while((child = getNestedException(parent)) != null) {
             // Get the child's message.
             String msg2 = child.getMessage();

             // If we found a message for the child exception,
             // we append it.
             if (msg2 != null) {
                 if (msg != null) {
                     msg += ": " + msg2;
                 } else {
                     msg = msg2;
                 }
             }

             // Any nested ApplicationException will append its own
             // children, so we need to break out of here.
             if (child instanceof ApplicationException) {
                 break;
             }
             parent = child;
         }

         // Return the completed message.
         return msg;
     }

     public void printStackTrace() {
         // Print the stack trace for this exception.
         super.printStackTrace();

         Throwable parent = this;
         Throwable child;

         // Print the stack trace for each nested exception.
         while((child = getNestedException(parent)) != null) {
             if (child != null) {
                 System.err.print("Caused by: ");
                 child.printStackTrace();

                 if (child instanceof ApplicationException) {
                     break;
                 }
                 parent = child;
             }
         }
     }

     public void printStackTrace(PrintStream s) {
         // Print the stack trace for this exception.
         super.printStackTrace(s);

         Throwable parent = this;
         Throwable child;

         // Print the stack trace for each nested exception.
         while((child = getNestedException(parent)) != null) {
             if (child != null) {
                 s.print("Caused by: ");
                 child.printStackTrace(s);

                 if (child instanceof ApplicationException) {
                     break;
                 }
                 parent = child;
             }
         }
     }

     public void printStackTrace(PrintWriter w) {
         // Print the stack trace for this exception.
         super.printStackTrace(w);

         Throwable parent = this;
         Throwable child;

         // Print the stack trace for each nested exception.
         while((child = getNestedException(parent)) != null) {
             if (child != null) {
                 w.print("Caused by: ");
                 child.printStackTrace(w);

                 if (child instanceof ApplicationException) {
                     break;
                 }
                 parent = child;
             }
         }
     }

     public Throwable getCause()  {
         return cause;
     }
}

The code in Listing 1 is fairly straightforward; we've simply chained together multiple exceptions to create a single, nested exception. The real benefit, however, is in using this technique as the starting point to create an application-specific hierarchy of exceptions. An exception hierarchy will let your EJB clients receive both business-specific exceptions and system-specific information, without requiring you to write a lot of extra code.

A hierarchy of exceptions

Your exception hierarchy should begin with something fairly robust and generic, like ApplicationException. If you make your top-level exception too specific you'll end up having to restructure your hierarchy later to fit in something more generic.

So, let's say that your application called for a NoSuchBookException, an InsufficientFundsException, and a SystemUnavailableException. Rather than create individual exceptions for each, you could set up each exception to extend ApplicationException, providing only the few additional constructors needed to create a formatted message. Listing 2 is an example of such an exception hierarchy:

Listing 2. An exception hierarchy

package com.ibm.library;

import com.ibm.ApplicationException;

public class NoSuchBookException extends ApplicationException {

     public NoSuchBookException(String bookName, String libraryName) {
     super("The book '" + bookName + "' was not found in the '" +
      libraryName + "' library.");
 }
}

The exception hierarchy makes things much simpler when it comes to writing numerous specialized exceptions. Adding a constructor or two for each exception class rarely takes more than a few minutes per exception. You will also often need to provide subclasses of these more specific exceptions (which are in turn subclasses of the main application exception), providing even more specific exceptions. For example, you might need an InvalidTitleException and a BackorderedException to extend NoSuchBookException.

Enterprise applications are often built with almost no attention given to exception handling. While it's easy -- and sometimes tempting -- to rely on low-level exceptions like RemoteException and NamingException, you'll get a lot more mileage out of your application if you start with a solid, well-thought-out exception model. Creating a nested, hierarchical exception framework will improve both your code's readability and its usability.

EJB best practices

2007-09-07T09:51:00.000-07:00

EJB best practices: Industrial-strength JNDI optimization

Use caching and a generic factory class to automate JNDI lookups
developerWorks

Integrate new tools and architectures into your environment -- fast!

Level: Intermediate

Brett McLaughlin (brett@newInstance.com), Author and Editor, O'Reilly Media Inc.

01 Sep 2002

Brett McLaughlin continues his EJB best practices with an examination of JNDI lookups, which are an essential and frequent part of almost all EJB interactions. Unfortunately, JNDI operations almost always exact a performance toll. In this tip, Brett shows you how a home-interface factory can reduce the overhead of JNDI lookups in your EJB applications.

Every kind of EJB component (session, entity, and message driven) has a home interface. The home interface is a bean's base of operations; once you've found it, you have access to that bean's functionality. EJB applications rely on JNDI lookups to access their beans' home interfaces. Because EJB apps tend to run multiple beans, and because JNDI lookups are often present in many components, much of an application's performance overhead can be spent on these lookups.

In this tip, we'll look at some of the most common JNDI optimizations. In particular, I'll show you how to combine caching and a generic helper class to create a factory-style solution to JNDI overhead.

Reducing context instances

Listing 1 shows a typical piece of EJB code, requiring multiple JNDI lookups. Study the code for a moment, and then we'll work on optimizing it for better performance.

public boolean buyItems(PaymentInfo paymentInfo, String storeName,
List items) {
// Load up the initial context
Context ctx = new InitialContext();

// Look up a bean's home interface
Object obj = ctx.lookup("java:comp/env/ejb/PurchaseHome");
PurchaseHome purchaseHome =
(PurchaseHome)PortableRemoteObject.narrow(obj, PurchaseHome.class);
Purchase purchase = purchaseHome.create(paymentInfo);

// Work on the bean
for (Iterator i = items.iterator(); i.hasNext(); ) {
purchase.addItem((Item)i.next());
}

// Look up another bean
Object obj = ctx.lookup("java:comp/env/ejb/InventoryHome");
InventoryHome inventoryHome =
(InventoryHome)PortableRemoteObject.narrow(obj, InventoryHome.class);
Inventory inventory = inventoryHome.findByStoreName(storeName);

// Work on the bean
for (Iterator i = items.iterator(); i.hasNext(); )
inventory.markAsSold((Item)i.next());
}

// Do some other stuff
}

While this example is somewhat contrived, it does reveal some of the most glaring problems with using JNDI. For starters, you might ask yourself if the new InitialContext object is necessary. It's likely that this context has already been loaded elsewhere in the application code, yet we've created a new one here. Caching the InitialContext instances would result in an immediate performance boost, as shown in Listing 2:

public static Context getInitialContext() {
if (initialContext == null) {
initialContext = new InitialContext();
}

return initialContext;
}

By using a helper class with the getInitialContext() instead of instantiating a new InitialContext for every operation, we've cut down the number of contexts floating around in our application to one.

Uh oh -- what about threading?
If you're worried about the effects of threading on the solution proposed here, don't be. It is absolutely possible that two threads could go to work on at the same time (thus creating two contexts at once) but this type of error would happen only on the first invocation of the method. Because the problem won't come up more than once, synchronization is unnecessary, and would in fact introduce more complexities than it would resolve.

Optimizing lookups

Caching the context instances is a step in the right direction, but we're not done optimizing yet. Every time we call the lookup() method it will perform a new lookup, and return a new instance of a bean's home interface. At least, that's the way JNDI lookups are usually coded. But wouldn't it be better to have just one home-interface per bean, shared across components?

Rather than looking up the home interface for PurchaseHome or InventoryHome again and again, we could cache each individual bean reference; that's one solution. But what we really want is a more general mechanism for caching home interfaces in our EJB applications.

The answer is to create a generic helper class to both obtain the initial context and look up the home interface for every bean in the application. In addition, this class should be able to manage each bean's context for various application components. The generic helper class shown in Listing 3 will act as a factory for EJB home interfaces:

package com.ibm.ejb;

import java.util.Map;
import javax.ejb.EJBHome;
import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;

public class EJBHomeFactory {

private static EJBHomeFactory instance;

private Map homeInterfaces;
private Context context;

// This is private, and can't be instantiated directly
private EJBHomeFactory() throws NamingException {
homeInterfaces = new HashMap();

// Get the context for caching purposes
context = new InitialContext();

/**
* In non-J2EE applications, you might need to load up
* a properties file and get this context manually. I've
* kept this simple for demonstration purposes.
*/
}

public static EJBHomeFactory getInstance() throws NamingException {
// Not completely thread-safe, but good enough
// (see note in article)
if (instance == null) {
instance = new EJBHomeFactory();
}
return instance;
}

public EJBHome lookup(String jndiName, Class homeInterfaceClass)
throws NamingException {

// See if we already have this interface cached
EJBHome homeInterface =
(EJBHome)homeInterfaces.get(homeInterfaceClass);
// If not, look up with the supplied JNDI name
if (homeInterface == null) {
Object obj = context.lookup(jndiName);
homeInterface =
(EJBHome)PortableRemoteObject.narrow(obj, homeInterfaceClass);

// If this is a new ref, save for caching purposes
homeInterfaces.put(homeInterfaceClass, homeInterface);
}
return homeInterface;
}
}

Inside the EJBHomeFactory class

The key to the home-interface factory is in the homeInterfaces map. The map stores each bean's home interface for use; as such, one home-interface instance can be used over and over again. You should also note that the key in the map is not the JNDI name passed into the lookup() method. It's quite common to have the same home interface bound to different JNDI names, but doing so can result in duplicates in your map. By relying on the class itself, you ensure that you won't end up with multiple home interfaces for the same bean.

Inserting the new home-interface factory class into the original code from Listing 1 will result in the optimized EJB lookup shown in Listing 4:

public boolean buyItems(PaymentInfo paymentInfo, String storeName,
List items) {

EJBHomeFactory f = EJBHomeFactory.getInstance();

PurchaseHome purchaseHome =
(PurchaseHome)f.lookup("java:comp/env/ejb/PurchaseHome",
PurchaseHome.class);
Purchase purchase = purchaseHome.create(paymentInfo);

// Work on the bean
for (Iterator i = items.iterator(); i.hasNext(); ) {
purchase.addItem((Item)i.next());
}

InventoryHome inventoryHome =
(InventoryHome)f.lookup("java:comp/env/ejb/InventoryHome",
InventoryHome.class);
Inventory inventory = inventoryHome.findByStoreName(storeName);

// Work on the bean
for (Iterator i = items.iterator(); i.hasNext(); ) {
inventory.markAsSold((Item)i.next());
}

// Do some other stuff
}

In addition to being more clear (at least in my opinion) the factory-optimized EJB lookup above will perform much faster over time. The first time you use the new class, you'll incur all the usual lookup penalties (assuming another portion of the application hasn't already paid them) but all future JNDI lookups should hum right along. It's also worth pointing out that the home-interface factory will not interfere with your container's bean management. Containers manage bean instances, not the home interfaces of those instances. Your container will still be in charge of instance swapping, as well as any other optimizations you want it to perform.

In the next installment of EJB best practices, I'll show you how you can enable administrative access to entity beans, without directly exposing them to your application's Web tier. Until then, I'll see you online.

JAVA Strengths

2007-09-05T09:07:00.000-07:00

JAVA Strengths

Java is an excellent programming language. For most programming it's better than older programming languages like C or C++.

Productivity The top reason Java has become popular is because of the increased productivity of Java programmers. It is claimed, and my experience is in agreement, that Java programmers have about double the productivity of C/C++ programmers.

GUI Java a good, portable library for a Graphical User Interface (GUI). Most programming languages supply only a text mode interface.

Internet Java lets you easily use the Internet. Most languages were designed before the Internet was born!

Portability Java programs can run on many different machines (Intel, Sparc, PowerPC, ...) and many different operating systems (Windows, Unix, Macintosh, ...). You can move a C program if it is written very carefully, but it is usually difficult or impossible. In contrast, it easy to move most Java programs.

Reliability Java programs are more reliable because Java doesn't have very dangerous things like C/C++'s pointer arithmetic. Java also checks array bounds and other error-prone operations. Memory management is much safer because Java does automatic garbage collection.

Libraries Java has a very large number of packages which extend the language. Therefore it is unnecessary to call the operating system directly.

OOP Object-oriented programming features (inheritance, encapsulation, and polymorphism) make many programs, especially large programs, easier to write.

Large Programs Java supports large programming projects with object-oriented programming, packages, and components (JavaBeans).

Buffer overflows likely to be around for another decade

2007-09-04T09:43:00.000-07:00

Buffer overflows likely to be around for another decade

By Edward Hurley, News Writer

Few classes of security flaws in software cause as many headaches as buffer overflows. Yet given the present programming paradigm, buffer overflows will be around for some time to come, experts say.

Buffer overflows in both Unix and Windows systems were common occurrences in the recently released SANS/FBI Top 20 List of Web vulnerabilities. Buffer overflows open gaping holes that attackers can exploit, as the recent Slapper worm showed. The worm took advantage of a buffer overflow in OpenSSL running on Linux Apache Web servers to set up peer-to-peer networks and to commandeer other computers for possible use in launching distributed denial-of-service attacks.

Often buffer overflow vulnerabilities are hard to find in the minutiae of computer code. Additionally, the nature of the popular C programming language makes them an easy programming error to make, experts said.

In essence, a buffer overflow occurs when too much data is stuffed into a memory space. They are common in applications written in C.

Gerhard Eschelbeck, vice president of engineering at Redwood Shores, Calif.-based Qualys Inc., compares buffer overflows to people filling in handwritten forms that allow one space for each letter of a person's name. A buffer overflow is similar to what happens when the writer does not having enough blocks for one's last name, he said.

The extra "letters" or data aren't lost but written into other places in memory. This could cause the application to act oddly or shut down, creating a denial-of-service condition. It can also allow an attacker to run malicious code on the system. "The code looks the same to the computer," Eschelbeck said.

Buffer overflows can occur in both the stack and heap areas of memory. Stack memory is a pool that all programs can share. When a buffer overflow occurs, the extra data is put into stack memory, much like a card is added to a deck, said Ted Doty, director of product management at Waltham, Mass.-based Okena Inc. The system doesn't realize that data may be malicious.

Buffer overflows in heap memory are similar, except that heap memory is specifically allocated to a program, Doty said.

Buffer overflows are hard to find, and that's why many make their way into finished software products. Basically, one has to squeeze large amounts of data into every area of a program that can accept it to find them, Doty said.

Once an attacker is able to find a buffer overflow, exploiting it is often quite simple.

Finding buffer overflows is especially hard in large server applications that may have as many as 40 million lines of code, Doty said. Software vendors are under too much pressure to get new versions out to market to audit every line of code, he added.

At their most basic, buffer overflow vulnerabilities are programming errors. Error-checking for a too-long string of data wasn't included in the program.

One reason for this is that the C programming language doesn't have boundary checking. It's assumed that someone programming in C would include error-checking in their code, Doty said.

Other languages such as Perl and Java have the utility, hence buffer overflows don't occur, Eschelbeck said. On the other hand, Perl and Java don't offer access to the system that is as deep as some programs need. "The C programming language is very powerful. You can go as deep as you need," he said.

Better programmer training and education would minimize buffer overflows. Certain C functions such as string copy need to be avoided in order to minimize buffer overflows. There are code-auditing tools that look for such flaws, but they are still pretty early in their development, Eschelbeck said.

Software users, however, need to keep on buffer overflows in the applications they run. This means keeping tabs on advisories released by the Computer Emergency Response Team at Carnegie Mellon University or by vendors. Installing the released patches is also a must.

While they occur in many applications, buffer overflows are most damaging in core infrastructure such as domain name servers and Web servers.

Even if all programs started today were written buffer-overflow free, the vulnerabilities will continue to be found for the next few years because of the millions of lines of code that have not been audited. "There's no doubt in my mind that we'll be dealing with buffer overflows for at least the next 10 years," Eschelbeck added.

Looking up an EJB home with JNDI

2007-09-03T08:24:00.000-07:00

Example: Looking up an EJB home with JNDI

Most applications that use JNDI run in a container. Some do not. The name used to look up an object depends on whether or not the application is running in a container. Sometimes it is more convenient for an application to use a corbaname URL as the lookup name. Container-based JNDI clients and thin Java clients can use a corbaname URL.

The following examples show how to perform JNDI lookups from different types of applications.

JNDI lookup from an application running in a container

Applications that run in a container can use java: lookup names. Lookup names of this form provide a level of indirection such that the lookup name used to look up an object is not dependent on the object's name as it is bound in the name server's name space. The deployment descriptors for the application provide the mapping from the java: name and the name server lookup name. The container sets up the java: name space based on the deployment descriptor information so that the java: name is correctly mapped to the corresponding object.

The following example shows a lookup of an EJB home. The actual home lookup name is determined by the application's deployment descriptors. The enterprise bean (EJB) resides in an EJB container, which provides an interface between the bean and the application server on which it resides.

// Get the initial context as shown in a previous example
...
// Look up the home interface using the JNDI name
try {
  java.lang.Object ejbHome =
     initialContext.lookup(
        "java:comp/env/com/mycompany/accounting/AccountEJB");
  accountHome = (AccountHome)javax.rmi.PortableRemoteObject.narrow(
     (org.omg.CORBA.Object) ejbHome, AccountHome.class);
}
  catch (NamingException e) { // Error getting the home interface
  ...
}

JNDI lookup from an application that does not run in a container

Applications that do not run in a container cannot use java: lookup names because it is the container which sets the java: name space up for the application. Instead, an application of this type must look the object up directly from the name server. Each application server contains a name server. System artifacts such as EJB homes are bound relative to the server root context in that name server. The various name servers are federated by means of a system name space structure. The recommended way to look up objects on different servers is to qualify the name so that the name resolves from any initial context in the cell. If a relative name is used, the initial context must be the same server root context as the one under which the object is bound. The form of the qualified name depends on whether the qualified name is a topology-based name or a fixed name. Examples of each form of qualified name follow.

Topology-based qualified names

Topology-based qualified names traverse through the system name space to the server root context under which the target object is bound. A topology-based qualified name resolves from any initial context in the cell.

Single server

The following example shows a lookup of an EJB home that is running in the single server, MyServer, configured in the node, Node1.

// Get the initial context as shown in a previous example
// Using the form of lookup name below, it doesn't matter which
// server in the cell is used to obtain the initial context.
...
// Look up the home interface using the JNDI name
try {
  java.lang.Object ejbHome = initialContext.lookup(
     "cell/nodes/Node1/servers/MyServer/com/mycompany/accounting/AccountEJB");
  accountHome = (AccountHome)javax.rmi.PortableRemoteObject.narrow(
     (org.omg.CORBA.Object) ejbHome, AccountHome.class);
}
catch (NamingException e) { // Error getting the home interface
  ...
}

Fixed qualified names

If the target object has a cell-scoped fixed name defined for it, you can use its qualified form instead of the topology-based qualified name. Even though the topology-based name works, the fixed name does not change with the specific cell topology or with the movement of the target object to a different server. An example lookup with a qualified fixed name is shown below.

// Get the initial context as shown in a previous example
// Using the form of lookup name below, it doesn't matter which
// server in the cell is used to obtain the initial context.
...
// Look up the home interface using the JNDI name
try {
  java.lang.Object ejbHome = initialContext.lookup(
    "cell/persistent/com/mycompany/accounting/AccountEJB");
  accountHome = (AccountHome)javax.rmi.PortableRemoteObject.narrow(
     (org.omg.CORBA.Object) ejbHome, AccountHome.class);
  }
catch (NamingException e) { // Error getting the home interface
...
}

JNDI lookup with a corbaname URL

A corbaname can be useful at times as a lookup name. If, for example, the target object is not a member of the federated name space and cannot be located with a qualifiied name, a corbaname can be a convenient way to look up the object. A lookup with a corbaname URL follows.

// Get the initial context as shown in a previous example
...
// Look up the home interface using a corbaname URL
try {
  java.lang.Object ejbHome = initialContext.lookup(
     "corbaname:iiop:someHost:2809#com/mycompany/accounting/AccountEJB");
  accountHome = (AccountHome)javax.rmi.PortableRemoteObject.narrow(
     (org.omg.CORBA.Object) ejbHome, AccountHome.class);
}
catch (NamingException e) { // Error getting the home interface
  ...
}

EJB and the limitation

2007-09-02T07:47:00.000-07:00

According to EJB specification, EJB can not directly use file. So we have to find a way to read/write file indirectly. There are some solution for this.
1. We can use RMI technology.
2. We can use JCA to read file.

Solution 1:

RMI is the way to read file. It is also easy to implemented. All we have to do is make a server which provide a service: read/write file. EJB will implement a client to communicate with this server. All data when read/write file between the EJB and RMi Server is in byte array. But one of the drawback is that RMI is so heavy and big.

Solution 2:
The J2EE Connector architecture is based on the technologies defined and standardized in the Java 2 Platform, Enterprise Edition (J2EE) and is part of the J2EE platform.

As more businesses move towards an e-business strategy, integration with existing enterprise information systems (EIS) becomes the key to success. Enterprises with successful e-businesses need to integrate their existing EIS systems with new web-based applications. They also need to extend the reach of their EIS systems to support business-to-business (B2B) transactions.

Before the the J2EE Connector architecture was defined, no specification for the Java platform addressed the problem of providing a standard architecture for integrating heterogeneous EIS systems. Most EIS vendors and application server vendors use non-standard vendor-specific architectures to provide connectivity between application servers and enterprise information systems. The following diagram illustrates the complexity of a heterogenous environment.

The Java 2 Platform, Enterprise Edition provides containers for client applications, web components based on Servlets and JavaServer Pages (JSP) technologies, and Enterprise JavaBeans (EJB) components. These containers provide deployment and runtime support for application components. They also provide a federated view of the services provided by the underlying application server for the application components.

Containers run on existing systems: web servers for the web containers, application servers, transaction processing (TP) monitors, and database systems for EJB containers. This enables enterprises to leverage both the advantages of their existing systems and those with J2EE technology. Enterprises can write (or rewrite) new applications using J2EE technology capabilities and can also encapsulate parts of existing applications with Enterprise JavaBeans or JavaServer Pages (JSP) technologies.

Enterprise applications access functions and data associated with applications running on EISs. Application servers extend their containers and support connectivity to heterogeneous EISs. Enterprise tools and Enterprise Application Integration (EAI) vendors add value by providing tools and frameworks to simplify the EIS integration task.

The J2EE Connector architecture defines a standard architecture for connecting the J2EE platform to heterogeneous EIS systems. Examples of EIS systems include ERP, mainframe transaction processing, database systems, and legacy applications not written in the Java programming language. By defining a a set of scalable, secure, and transactional mechanisms, the J2EE Connector architecture enables the integration of EISs with application servers and enterprise applications.

The J2EE Connector architecture enables an EIS vendor to provide a standard resource adapter for its EIS. The resource adapter plugs into an application server, providing connectivity between the EIS, the application server, and the enterprise application. If an application server vendor has extended its system to support the J2EE Connector architecture, it is assured of seamless connectivity to multiple EISs. An EIS vendor needs to provide just one standard resource adapter which has the capability to plug in to any application server that supports the J2EE Connector architecture.

Multiple resource adapters (that is, one resource adapter per type of EIS) are pluggable into an application server. This capability enables application components deployed on the application server to access the underlying EIS systems.

Now all we have is implemented ResourceAdapter to provide a File Service to EJB. It is so tricky. But it works.

How to validate JAXP 1.3

2007-08-24T22:17:00.000-07:00

How to use JAXP to validate a xml file against schema.
File NewXMLSchema.xsd is a resource.
File NewXMlSchema.xml is a xml file which we need to validate against schema NewXMlSchema.xsd

package validator;

import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.net.URL;

import javax.xml.XMLConstants;
import javax.xml.transform.stream.StreamSource;
import javax.xml.validation.Schema;
import javax.xml.validation.SchemaFactory;
import javax.xml.validation.Validator;

import org.xml.sax.ErrorHandler;
import org.xml.sax.SAXException;
import org.xml.sax.SAXParseException;

public class JAXPValidation {
public JAXPValidation() {
try {
SchemaFactory factory = SchemaFactory.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI);
Schema schema = factory.newSchema(this.getClass().getResource("NewXMLSchema.xsd"));
Validator validator = schema.newValidator();
validator.setErrorHandler(new ErrorHandler(){

public void error(SAXParseException exception) throws SAXException {
exception.printStackTrace();
throw exception;
}

public void fatalError(SAXParseException exception) throws SAXException {
exception.printStackTrace();
throw exception;
}

public void warning(SAXParseException exception) throws SAXException {
exception.printStackTrace();
throw exception;
}
});
URL url = this.getClass().getResource("NewXMlSchema.xml");
InputStream stream = url.openStream();
StreamSource source = new StreamSource(stream);
validator.validate(source);
System.out.println("validate sucessfully");
} catch (SAXException e) {
e.printStackTrace();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}

static public void main(String[] args) {
new JAXPValidation();
}
}

How to validate request in jax-ws

2007-08-21T08:01:00.000-07:00

JAX-WS has the hole in validate soap request. But we can work around by using SoapHandler and JAXP to validate soap request.
As we can see from the picture from the left, we have 2 chance to validate request:
SOAPHandler and LogicalHandler. We use SOAPHandler if we want to validate soap header and soap body and use logicalhandler to validate soap body.

In method: handleMessage() in class SOAPHanlder, we can easily get the raw soap message by using: SOAPMessageContext and write them to ByteArrayOutputStream. After we got the xml string, we can use JAXP to validate the request. Detail in how to implement the idea will be presented in next post.

Default Password List for Routers

2007-08-19T09:13:00.000-07:00

To know the default password list for router, please refer to this url:

http://www.phenoelit-us.org/dpl/dpl.html

How to deploy an FTP server with FileZilla

2007-08-16T08:56:00.000-07:00

Today, i meet a problem with ftp server. I will share my experience with you about it.

Download FTP Server at:
http://filezilla.sourceforge.net/
Download FTP Client at:
http://filezilla.sourceforge.net

Setup FileZilla server:
1.Run installer.
2.If you are behind the router, remember to open 2 port at router: port 20, 21.
By opening these 2 ports, the users from internet can access to server. If your server is behind the firewall, you have to let firewall also open 2 ports 20,21 .
3. Next, please refer to this document:
http://filezilla.sourceforge.net/documentation/

Hope this post will help you in setup ftp server: FileZilla.

Java classloader

2007-07-21T21:53:00.001-07:00

The Java platform was designed to be robust, secure, and extensible in order to support the mobility of code and data. The Java ClassLoader in the Java Virtual Machine (JVM) is a key component in the realization of these goals.

The JVM is responsible for loading and executing code on the Java platform. It uses a ClassLoader to load Java classes into the Java runtime environment. ClassLoaders are architected so that at start-up the JVM doesn't need to know anything about the classes that will be loaded at runtime. Almost all Java-based containers such as EJB or servlet containers implement custom ClassLoaders to support features like hot deployment and runtime platform extensibility. An in-depth understanding of ClassLoaders is important for developers when implementing such Java-based containers.

For enterprise developers who develop components that are deployed on these containers, this knowledge will help you understand how the container works and with debugging problems. This article presents the Java ClassLoader architecture and discusses the implications of ClassLoaders on platform security and extensibility as well as a method to implement user-defined ClassLoaders.

The smallest unit of execution that gets loaded by a ClassLoader is the Java class file. A class file contains the binary representation of a Java class, which has the executable bytecodes and references to other classes used by that class, including references to classes in the Java API. Stated simply, a ClassLoader locates the bytecodes for a Java class that needs to be loaded, reads the bytecodes, and creates an instance of the java.lang.Class class. This makes the class available to the JVM for execution. Initially when a JVM starts up, nothing is loaded into it. The class file of the program being executed is loaded first and then other classes and interfaces are loaded as they get referenced in the bytecode being executed. The JVM thus exhibits lazy loading, i.e., loading classes only when required, so at start-up the JVM doesn't need to know the classes that would get loaded during runtime. Lazy loading plays a key role in providing dynamic extensibility to the Java platform. The Java runtime can be customized in interesting ways by implementing a custom ClassLoader in a program, as I'll discuss later.

Java 2 ClassLoader Delegation Model
Multiple instances of ClassLoaders exist in the Java 2 runtime, each loading classes from different code repositories. For instance, Java core API classes are loaded by the bootstrap (or primordial) ClassLoader. Application-specific classes are loaded by the system (or application) ClassLoader. In addition, an application can define its own ClassLoaders to load code from custom repositories. Java 2 defines a parent-child relationship between ClassLoaders. Each ClassLoader except the bootstrap ClassLoader has a parent ClassLoader, conceptually forming a treelike structure of ClassLoaders. The bootstrap ClassLoader is the root of this tree and thus doesn't have a parent. The relationship is depicted in Figure 1.

The following is a high-level class-loading algorithm executed by a ClassLoader when a client requests it to load a class:
1. A check is performed to see if the requested class has already been loaded by the current ClassLoader. If so, the loaded class is returned and the request is completed. The JVM caches all the classes that are loaded by a ClassLoader. A class that has previously been loaded by a ClassLoader is not loaded again.
2. If the class is not already loaded, the request is delegated to the parent ClassLoader, before the current ClassLoader tries to load it. This delegation can go all the way up to the bootstrap ClassLoader, after which no further delegation is possible.
3. If a parent fails to return a class because it was unable to load it, the current ClassLoader will then try to search for the requested class. Each ClassLoader has defined locations where it searches for classes to load. For instance, the bootstrap ClassLoader searches in the locations (directories and zip/jar files) specified in the sun.boot.class.path system property. The system ClassLoader searches for classes in the locations specified by the classpath (set as the java.class.path system property) command-line variable passed in when a JVM starts executing. If the class is found, it's loaded into the system and returned, completing the request.
4. If the class is not found, a java.lang.ClassNotFoundException is thrown.

Implementing a Java 2 Custom ClassLoader
As mentioned earlier the Java platform allows an application programmer to customize the classloading behavior by implementing a custom ClassLoader. This section shows how.

ClassLoaders (all but the bootstrap ClassLoader, which is implemented in native code in the JVM) are implemented by extending the java.lang.Class Loader class. The following code shows the relevant methods of the Java 2 ClassLoader API:

1. public abstract class ClassLoader extends Object {
2. protected ClassLoader(ClassLoader parent);
3. protected final Class defineClass(
4. String name,byte[] b,int off,int len)
5. throws ClassFormatError
5. protected Class findClass(String
7. className) throws ClassNotFoundException
6. public class loadClass(String className)
7. throws ClassNotFoundException
8.}

Each ClassLoader is assigned a parent when it's created, as per the parent-delegation model. Clients invoke the loadClass method on an instance of a ClassLoader to load a class. This initiates the classloading algorithm as explained earlier. Prior to Java 2, the loadClass method in the java.lang.ClassLoader class was declared abstract, requiring custom ClassLoaders to implement it when extending the java.lang.ClassLoader class. Implementing the loadClass method is rather complicated, so this has been changed in Java 2. With the introduction of the ClassLoader parent delegation model, java.lang.ClassLoader has an implementation of the loadClass method, which is essentially a template method that executes the classloading algorithm. The loadClass method invokes the findClass method (introduced in Java 2) in Step 3 of the classloading algorithm. Custom ClassLoaders should override this method to provide a custom way of locating and loading a Java class. This greatly simplifies the implementation of a custom ClassLoader. Listing 1 shows some code from the CustomClassLoader.java class (the complete source code can be downloaded from www.sys-con.com/java/sourcec.cfm), which loads classes from a repository specified in the constructor.

The findClass method invokes loadFromCustomRepository that searches for the given class in the repository and, if found, reads and returns the bytecodes for the class. The raw bytecodes for the class are passed into the defineClass method implemented in the java.lang.ClassLoader class, which returns an instance of the java.lang.Class object. This makes a new class available to a running Java program. The defineClass method also ensures that a custom ClassLoader does not redefine core Java API classes by loading them from a custom repository. A SecurityException is thrown if the class name passed to defineClass begins with "java".

It should be noted that at start-up, the JVM doesn't need to know anything about the class represented by the string passed into the loadClass method. A subsequent section shows how a program can use the CustomClassLoader.

Deviations from the Java 2 Delegation Model
The Java 2 delegation model cannot be followed in all situations. There are cases in which ClassLoaders have to diverge from the Java 2 model. For instance, the servlet specification recommends (section 9.7) that a Web application ClassLoader be implemented so that classes and resources packaged in the Web application archive are loaded in preference to classes and resources residing in container-wide JAR files. To meet this recommendation, a Web application ClassLoader should search for classes and resources in its local repository first, before delegating to a parent ClassLoader, thus deviating from the Java 2 delegation model. This recommendation makes it possible for Web applications to use different versions of classes/resources than those being used by a servlet container. For example, a Web application might be implemented using features available in a newer version of an XML parser than the one being used by a servlet container.

A Web application ClassLoader that meets the recommendation of the servlet specifications can be implemented by overriding the loadClass method of the java.lang.Classloader class. The loadClass method of such a custom ClassLoader may look similar to Listing 2.

Applications of ClassLoaders
ClassLoaders provide some powerful features that can be utilized in Java programs. This section discusses a few ways in which they can be used.

Hot Deployment
Upgrading software in a running application without restarting it is known as hot deployment. For a Java application, hot deployment means upgrading Java classes at runtime. ClassLoaders play an important role in Java-based application servers to achieve hot deployment. This feature is exploited in most Java-based application servers like EJB servers and servlet containers. A ClassLoader cannot reload a class that it has already loaded, but using a new instance of a ClassLoader will reload a class into a running program. The following code from TestCustomLoader.java illustrates how hot deployment may be achieved in a Java application:

1. ClassLoader customLoader = new
2. CustomClassLoader(repository);
3. loadAndInvoke(customLoader,classToLoad);
4. System.out.println("waiting.Hit
5. Enter to continue");
6. System.in.read();
7. customLoader = new CustomClassLoader
8. (repository);
9. loadAndInvoke(customLoader,classToLoad);

An instance of the CustomClassLoader is created to load classes from a repository specified as a command-line parameter. loadAndInvoke loads a class, HelloWorld, also specified as a command-line parameter, and invokes a method on its instance, which prints a message on the console. While the program is waiting for user input at line 6, the HelloWorld class can be changed (by changing the message that gets printed on the console) and recompiled. When the program continues execution, a new instance of CustomClassLoader is created at line 7. When loadAndInvoke executes line 9, it loads the updated version of HelloWorld and a new message is printed on the console.

Modifying Class Files
A ClassLoader searches for bytecodes of a class file in the findClass method. After the bytecodes have been located and read into the program, they may be modified before invoking defineClass. For example, extra debugging information may be added to the class file before invoking defineClass. Class file data for some secure applications may be stored encrypted in a repository; the findClass method can decrypt the data before invoking defineClass. A program can generate the bytecodes on the fly instead of retrieving them from a repository. This forms the basis of JSP technology.

ClassLoaders and Security
Since a ClassLoader is responsible for bringing code into the JVM, it's architected so that the security of the platform may not be compromised. Each ClassLoader defines a separate namespace for the classes loaded by it, so at runtime a class is uniquely identified by its package name and the ClassLoader that loaded it. A class is not visible outside its namespace; at runtime there's a protective shield between classes existing in separate namespaces. The parent delegation model makes it possible for a ClassLoader to request classes loaded by its parent, thus a ClassLoader doesn't need to load all classes required by it.

The various ClassLoaders that exist in a Java runtime have different repositories from which they load code. The idea behind separating repository locations is that different trust levels can be assigned to the repositories. The Java runtime libraries loaded by the bootstrap ClassLoader have the highest level of trust in the JVM. The repositories for user-defined ClassLoaders have lower levels of trust. Furthermore, ClassLoaders can assign each loaded class into a protection domain that defines the permissions assigned to the code as it executes. To define permissions on code based on the system security policy (an instance of java.security.Policy), a custom ClassLoader should extend the java.security.SecureClassLoader class and invoke its defineClass method that takes a java.security.CodeSource object as a parameter. The defineClass method of SecureClassLoader gets the permissions associated with the CodeSource from the system policy and defines a java.security.Protection Domain based on that. A detailed discussion of the security model is beyond the scope of this article. Further details can be obtained from the book Inside the Java Virtual Machine by Bill Venners.

Summary
ClassLoaders offer a powerful mechanism by which the Java platform can be extended in interesting ways at runtime. Custom ClassLoaders can be used to achieve functionality not normally available to a running Java program. Some of these applications have been discussed in this article. ClassLoaders play an important role in some of the technologies offered by current J2EE platforms. For further details about the Java classloading mechanism, read Inside the Java Virtual Machine.

How to create rss link for google blogspot ?

2007-07-20T20:00:00.000-07:00

RSS Feeds from Xanga and Blogspot

Recently got a few friends who are blogging on Xanga and Blogspot. As someone who likes to read blog yet lazy to visit them one by one, I usually just subscribe to their RSS and wait for news to arrive at my doorstep (or better know as Outlook). However, I cannot find any [XML] or [RSS] icon in both the blog sites. I quick search in goggle reveals 2 ways which I can feed the blogs

Xanga:

there is a free service @
http://www.ephemeraleuphoria.com/xanga/rss.php?username=

Simply change the field to the username of the blog you wish to feed from Xanga

Blogspot:

Blogspot default feed through atom, which the RSS aggregator I using (you subscribe) do not support.
Therefore I used this free service from 2rss to convert the feeds to atom before subscibing to them

http://www.2rss.com/software.php?page=atom2rss

Lastly, there are some who simply dun support RSS feeds. I therefore use watchthatpage instead, which is recommended by icelava

JAXB tutorial and tip

2007-07-13T18:40:00.000-07:00

JAXB is used for mapping xml to java classes.

To generate java class from xml we use xjc.

Command: xjc -d sourcefolder schema.

Tip: Do not try to use parameter: -p in command line. This will make all class in the same package.
It results in namespace will be removed because of JAXB optimization. I've got a problem because it.

Unmarshalling:
JAXBContext context = JAXBContext.newInstance("enter generated packagename here");
Unmarshaller unmarshaller = context.createUnmarshaller();
unmarshaller.unmarshal(new FileInputStream(fileName));

Marshalling:
JAXBContext context = JAXBContext.newInstance("enter generated packagename here");
JAXBElement root = soapFactory.createEnvelope(rootElement);
Marshaller marshaller = context.createMarshaller();
marshaller.marshal(root, System.out);

How to configure pom file to use jaxws library and wsimport plugins maven

2007-07-07T08:26:00.000-07:00

First: you should point out the plugin repository to download jaxws-maven-plugin

[pluginRepository]
Repository url: http://download.java.net/maven/2
[/pluginRepository]

Second:
Chose the repository for downloading jaxws library:

[repository]
[id]java net repository[/id]
[name]java net name[/name]
[url]https://maven-repository.dev.java.net/repository[/url]
[layout]legacy[/layout]
[/repository]

Third:
Chose the plugin jaxws-maven-plugin

[plugin]
[groupId]org.codehaus.mojo[/groupId]
[artifactId>jaxws-maven-plugin[/artifactId]
[version]1.3[/version]
[executions]
[execution]
[goals]
[goal]wsimport[/goal]
[/goals]
[/execution]
[/executions]
[configuration]
[sourceDestDir]
${basedir}/src/main
[/sourceDestDir]
[verbose]true[/verbose]
[packageName]com.example.myschema[/packageName]
[target]2.1[/target]
[/configuration]

[dependencies]
[dependency]
[groupId]com.sun.xml.ws[/groupId]
[artifactId>jaxws-tools[/artifactId[
[version]2.1.1[/version]
[/dependency]
[dependency]
[groupId]javax.xml.bind[/groupId]
[artifactId]jaxb-api[/artifactId]
[version] 2.1[/version]
[/dependency]
[/dependencies]
[/plugin]

Configure plugins jaxws-maven-plugin with jdk1.6

2007-07-07T08:16:00.000-07:00

Today, i will write about how to use plugins jaxws-maven-plugin with jdk 1.6.
The prolem is : JDK 1.6 shipped with jaxws 2.0. But if you want to use jaxws-maven-plugin with latest jaxws library 2.1, you will meet the error:

JAXB 2.0 API is being loaded from the bootstrap classloader, but this RI (from jar:file:/C:/Documents and Settings/Administrator.HOME/.m2/repository/com/sun/xml/bind/jaxb-impl/2.1.3/jaxb-impl-2.1.3.jar!/com/sun/xml/bind/v2/model/impl/ModelBuilder.class) needs 2.1 API. Use the endorsed directo
ry mechanism to place jaxb-api.jar in the bootstrap classloader. (See http://java.sun.com/j2se/1.5.0/docs/guide/standards/)

This error occurred because jaxb 2.0 was loaded instead of your jaxb 2.1.

Solution:
First: you should find where is your java-home.
Second: go to ${java-home}/jre/lib and create folder: endorsed.
In this folder(endorsed) you copy your jaxb-api and jaxws-api (version 2.1 not 2.0)

Now done!
Test: mvn jaxws:wsimport and see the result.

Elipse and SVN

2007-07-01T22:59:00.000-07:00

Subclipse is the plugin for eclipse. This plugin provides eclipse the SVN capability.

Home page for subclipse:
http://subclipse.tigris.org/

Eclipse update site URL: http://subclipse.tigris.org/update_1.2.x
We use this URL for update plugin automatically.

Maven tutorial

2007-07-01T22:16:00.000-07:00

Today, i will make a tut about maven.

Definition about maven:

Maven was originally started as an attempt to simplify the build processes in the Jakarta Turbine project. There were several projects each with their own Ant build files that were all slightly different and JARs were checked into CVS. We wanted a standard way to build the projects, a clear definition of what the project consisted of, an easy way to publish project information and a way to share JARs across several projects.

The result is a tool that can now be used for building and managing any Java-based project. We hope that we have created something that will make the day-to-day work of Java developers easier and generally help with the comprehension of any Java-based project.

Home page: http://maven.apache.org/index.html

Step 1: Download maven from home page.

Step 2: configure setting.xml in folder maven/conf

Tag localRepository : is used to set the folder where maven will downloaded all library and find them when build project.

Tag Proxy: is used when you are behind the network proxy.

Step 3: Set up project layout.

After create the folder for our project. We have to create pom file for them.
At the folder project. We have a top pom file pom.xml.
At the folder modules, we also have a pom.xml file.
At module1, module2, we also have a pom.xml file.

Using tag modules and module to show which one will be compiled next.

[modules]
[module] modules [/module]
[/modules]

Using repository tag to show which repository maven used to get library:

[repositories]
[repository]

[/repository]
[/repositories]

Using dependency tag to show which library you used in the project.
[dependencies]
[dependency]
[/dependency]
[/dependencies]

JAX-WS and how to send SOAP Header

2007-06-05T09:08:00.001-07:00

When i do my project, i met a problems. How can i add soap header to soap messages. The webservice is developed with JAX-WS library.

I really don't know why jax-ws does not implements such a method to add soap header from the implementation of web-service. We have to use another way to do that: Soap Handler. A Soap Handler is called before the implementation of webservice called. So we used them to send soap header.

for example:
we have a webservice with one function: sayHello(). They function required that its response and request come along with a soap header. It should contains the value of session.

[soap:header]
[session] 12424512[/session]
[/soap:header]
[soap:body]
[sayhelloresponse] .... [/sayhelloresponse]
[/soap:body]

How we add soap:header??

Step to do:
in the class implement the function sayHello()

@Resource
WebServiceContext wsContext

void sayHello(){
MessageContext message = wsContext.getMessage();
message.put("session", 245235); //this will put the value to the context of the //webservice
}

In the soap handler, we get the value we store in the context of webservice. and put to soap handler. Only in soap handler, we have a chance to create soap header. So that why we have to do in this way.

public class SOAPLoggingHandler implements
SOAPHandler {
...

public boolean handleMessage(SOAPMessageContext smc) {
Boolean outboundProperty = (Boolean)
smc.get(MessageContext.MESSAGE_OUTBOUND_PROPERTY);

if (outboundProperty) {
MessageContext context = smc.getMessageContext();;
String session = context.get("Session");
SOAPMessage soapMessage = smc.getMessage();
//Add header to soapmessage. Using function of soap message like
//GetElementHeader, GetHeader ... I am not remember exactly. I will update this soon.
}
return true;
}

public boolean handleFault(SOAPMessageContext smc) {
logToSystemOut(smc);
return true;
}

public void close(MessageContext messageContext) {
}
}

JBoss and Log4j

2007-06-01T23:44:00.000-07:00

JBoss and Log4j come together will make the stupid things. JBoss make all logs of user application come to its log file and prohibit the user using trace log. It is terrible. In my project, i have to define my own log file which separated with the system log. But this task take me 2 days. What a terrible. Now i find 2 solution.
First one: Remove all the trace log and define new appender and category in log4j.xml in folder conf of jboss. It will make the log of application come to new place.
Seconde one:
Using RepositorySelector with ServletListener or Listener to use your own log. As my research in internet, i found that this is the clean solution of separate logger with jboss logger.
All you need for the second solution is:

Log4j.jar in WEB-INF.
Your own lof4j.xml file( you can mimic the log4j.xml of jboss)
2 file followings.
Configure web.xml to ensure servlet or servlet listener to run first to make log initialize or you will get error.

/***************************************
* *
* JBoss: The OpenSource J2EE WebOS *
* *
* Distributable under LGPL license. *
* See terms of license at gnu.org. *
* *
***************************************/

package org.jboss.repositoryselectorexample;

import java.io.InputStream;
import java.util.HashMap;
import java.util.Map;
import javax.servlet.ServletConfig;
import javax.servlet.ServletException;
import javax.xml.parsers.DocumentBuilderFactory;
import org.apache.log4j.Hierarchy;
import org.apache.log4j.Level;
import org.apache.log4j.LogManager;
import org.apache.log4j.spi.LoggerRepository;
import org.apache.log4j.spi.RepositorySelector;
import org.apache.log4j.spi.RootCategory;
import org.apache.log4j.xml.DOMConfigurator;
import org.w3c.dom.Document;

/**
* This RepositorySelector is for use with web applications. It assumes that
* your log4j.xml file is in the WEB-INF directory.
*
* @author Stan Silvert
*/
public class MyRepositorySelector implements RepositorySelector
{
private static boolean initialized = false;
private static Object guard = LogManager.getRootLogger();

private static Map repositories = new HashMap();
private static LoggerRepository defaultRepository;

/**
* Register your web-app with this repository selector.
*/
public static synchronized void init(ServletConfig config)
throws ServletException {
if( !initialized ) // set the global RepositorySelector
{
defaultRepository = LogManager.getLoggerRepository();
RepositorySelector theSelector = new MyRepositorySelector();
LogManager.setRepositorySelector(theSelector, guard);
initialized = true;
}

Hierarchy hierarchy = new Hierarchy(new RootCategory(Level.DEBUG));
loadLog4JConfig(config, hierarchy);
ClassLoader loader = Thread.currentThread().getContextClassLoader();
repositories.put(loader, hierarchy);
}

public static synchronized void removeFromRepository() {
repositories.remove(Thread.currentThread().getContextClassLoader());
}

// load log4j.xml from WEB-INF
private static void loadLog4JConfig(ServletConfig config,
Hierarchy hierarchy)
throws ServletException {
try {
String log4jFile = "/WEB-INF/log4j.xml";
InputStream log4JConfig =
config.getServletContext().getResourceAsStream(log4jFile);
Document doc = DocumentBuilderFactory.newInstance()
.newDocumentBuilder()
.parse(log4JConfig);
DOMConfigurator conf = new DOMConfigurator();
conf.doConfigure(doc.getDocumentElement(), hierarchy);
} catch (Exception e) {
throw new ServletException(e);
}
}

private MyRepositorySelector() {
}

public LoggerRepository getLoggerRepository() {
ClassLoader loader = Thread.currentThread().getContextClassLoader();
LoggerRepository repository = (LoggerRepository)repositories.get(loader);

if (repository == null) {
return defaultRepository;
} else {
return repository;
}
}
}

/***************************************
* *
* JBoss: The OpenSource J2EE WebOS *
* *
* Distributable under LGPL license. *
* See terms of license at gnu.org. *
* *
***************************************/

package org.jboss.repositoryselectorexample;

import java.io.*;
import java.net.*;

import javax.servlet.*;
import javax.servlet.http.*;
import org.apache.log4j.Logger;
import org.w3c.dom.Document;
import org.w3c.dom.Element;

/**
* This servlet demonstrates initialization of a RepositorySelector so that
* logging can be done on a per-app basis.
*
* @author Stan Silvert
*/
public class Log4JServlet extends HttpServlet {

private static Logger LOG;

private String servletName;
private MyObject myObj; // can't initialize until init() method called

/** Initializes the servlet.
*/
public void init(ServletConfig config) throws ServletException {
super.init(config);
this.servletName = config.getServletName();

MyRepositorySelector.init(config);

// note that we can't call Logger.getLogger() until
// MyRepositorySelector.init() is called. For all other classes in the
// webapp, you can call Logger.getLogger() at any time.
this.LOG = Logger.getLogger(Log4JServlet.class);

// the same goes for the Logger.getLogger() method in MyObject
this.myObj = new MyObject(this.servletName);
}

/** Destroys the servlet.
*/
public void destroy() {
MyRepositorySelector.removeFromRepository();
}

/**
* Processes requests for both HTTP GET and POST
* methods.
* @param request servlet request
* @param response servlet response
*/
protected void processRequest(HttpServletRequest request,
HttpServletResponse response)
throws ServletException, IOException {

response.setContentType("text/*");
PrintWriter out = response.getWriter();

LOG.error("TEST FROM THE SERVLET: " + this.servletName);
myObj.foo();
out.println("");
out.close();
}

/** Handles the HTTP GET method.
* @param request servlet request
* @param response servlet response
*/
protected void doGet(HttpServletRequest request,
HttpServletResponse response)
throws ServletException, IOException {
processRequest(request, response);
}

/** Handles the HTTP POST method.
* @param request servlet request
* @param response servlet response
*/
protected void doPost(HttpServletRequest request,
HttpServletResponse response)
throws ServletException, IOException {
processRequest(request, response);
}

/** Returns a short description of the servlet.
*/
public String getServletInfo() {
return "Log4JServlet";
}

}

/***************************************
* *
* JBoss: The OpenSource J2EE WebOS *
* *
* Distributable under LGPL license. *
* See terms of license at gnu.org. *
* *
***************************************/

package org.jboss.repositoryselectorexample;

import org.apache.log4j.Logger;

/**
* This just shows another object besides the servlet within the same
* web app.
*
* @author Stan Silvert
*/
public class MyObject {

// note that this can not be static because MyObject is referenced in
// the servlet before the RepositorySelector is initialized. For other
// classes, it would be OK to make this static.
private final Logger LOG = Logger.getLogger(MyObject.class);

private String servletName;

public MyObject(String servletName) {
this.servletName = servletName;
}
public void foo() {
LOG.info("Message from MyObject: " + this.servletName);
}
}

JMock and how to use.

2007-06-01T23:18:00.000-07:00

Today i use JMock for my projects. I should say that it is a good tool for testing.
But finding how to use it makes me crazy. But now i have just resolved all my problems.
It related to the jax-ws mock object. JAX-WS uses the abstract instead of the interface so
it makes using jmock is difficult task. But thanks for new lib JMock 2.x, it support abstract class also concrete class. Following is sample of using jmock for jax-ws soap handler for the unit test purpose.

Step to use:
Using the latest lib JMock 2.x lib

Import org.jmock.Expectations;
import org.jmock.integration.junit3.MockObjectTestCase;
import org.jmock.lib.legacy.ClassImposteriser;

protected final void setUp() throws Exception {
super.setUp();
setImposteriser(ClassImposteriser.INSTANCE);
soapHandlerImpl = new SoapHandlerImpl();
}

public final void testHandleMessage(){
//create mock object
final SOAPMessageContext smc = mock(SOAPMessageContext.class);
final SOAPHeader mockSoapHeader = mock(SOAPHeader.class);
final SOAPMessage mockSoapMessage1 = mock(SOAPMessage.class);
final SOAPMessage mockSoapMessage2 = mock(SOAPMessage.class);
final SOAPHeaderElement mockSoapHeaderElement =
mock(SOAPHeaderElement.class);
final SOAPHeaderElement soapSessionIDHashTag =
mock(SOAPHeaderElement.class);
final SOAPHeaderElement soapFromTag = mock(SOAPHeaderElement.class);
final SOAPHeaderElement soapFromTo = mock(SOAPHeaderElement.class);

//Make the expectation. Please remember that the expectation have to be
//correct or the exception occur. The exception message is not useful to resolve
checking(new Expectations() {
{
one(smc).get(MessageContext.MESSAGE_OUTBOUND_PROPERTY);
will(returnValue(true));
}
}); //simulate function getmessage for soapmessage.

//call function to test with mock parameter.
soapHandlerImpl.handleMessage(smc) ;
}

Some note:
To make the test run with out exception, we have to define how many mock object you need in the function. You should also count the number of function of mock object which will be called and the data for each calls. All of them have to be explicit define or you will got an exception in unit test.
I wrote this to remind me the problems when i do my project in company. I met a lot of prolems with that.

sitemap

2007-05-16T10:30:00.001-07:00

http://thegioitinhoccuatoi.blogspot.com/

Study and Works

2007-05-16T10:16:00.000-07:00

Now i am very tired because i have to study and work full time at the same time. Stress !!!! Work very hard but money is so little. I know people in in USA which do same job have the salary greater 10 times than mine. That is why i have to work more to find out changes to get higher salary and position. Studying at university. It required that i have to spent many time for research. Now i wished a day have 48hours. :D.

Tryyyyyyyyyy your best! I think i will be succeeded.

Vertu -- The most expesive phone

2007-05-16T09:22:00.000-07:00

Vertu Diamond Cellphone is the world most expensive phone, with a price tag of 1 million Euro, only 3 of these made are available. For the moment, only 2 has been grabbed, one by a russian tycoon and the other by a Hong Kong tycoon. The “Piece Unique” handmade phone is constructed out of solid gold, is studded with diamonds and comes with a customized insurance policy. The phone has entered the Guinness Book of World Records as the expensive cell phone ever sold in the world. One million euro for a phone is crazy, some people ARE crazy!

Nokia N95 -- the modern and expensive phone

2007-05-16T09:09:00.000-07:00

Wiith HSDPA technology which is 3 times faster than the 3G technology, Nokia N95 is the most prolific mobile phone from Nokia. The mobile phone is a camera mobile phone with imaging capability of 5 mega pixels. The sharpness and richness of colours brings something extra that other mobile phones cannot derive.
The most astounding feature of Nokia N95 is the technology that comes with it. In addition, the powerful camera which is included in the mobile phone gives you an imaging capability other mobile phones cannot give. The internal memory size of the mobile phone too is of great comfort to those who wish to store more images, video or music. Technology which runs Nokia N95 is S60 software on Symbian OS. Along with the various possibilities like mobile blogging, direct connection to TV, video calls, visual radio and faster internet connectivity this mobile phone brings out the best experience a latest mobile phone can give you.

Built-in GPS is another feather in the cap in Nokia N95. Inclusion of this technology in the mobile phone lets you find help in distress. Whether you wish to play Java™ enabled games, listen and watch stream music, all can be done with the aid of this mobile phone. As the mobile phone support a number of network types you can carry this around anywhere in the world and still stay connected. For connectivity and sharing Nokia N95 comes with latest technology. It is enabled with UPnP, Integrated wireless LAN, IrDA and Bluetooth™ technology which makes it universally acceptable in connectivity and sharing of data.

Apple iPhone

2007-05-16T09:00:00.000-07:00

With screen size of 3.5 inches and resolution of 320×480 at 160 ppi, Apple iPhone is a communication device sporting some of the latest technology. The weight of this device is 135 grams and overall dimensions are 115×61×11.6 mm. Included in this device is a 2.0 megapixels digital camera. The input method supported by Apple iPhone is multi-touch technology.

Apple iPhone

Dimensions : 104 x 50 x 16.4 mm
Weight : 93 g
Stand-by: 264 hrs

The operating system that runs Apple iPhone is OS X from Apple. This communication device supports wireless data transfer with the usage of technologies like Wi-Fi, EDGE and Bluetooth 2.0. It is a quad-band mobile phone supporting GSM 850/900/1800/1900 MHz. The storage capacity of Apple iPhone is 4GB or 8GB depending upon the need choice of the user.

With talk time of up to 5 hours and audio playback time of up to 16 hours, this is a communication device, which would let a user enjoy these functionalities too. Apple iPhone would be sporting a full QWERTY keyboard, which would let a user type in messages or emails easily. In addition, the keyboard would be supported by predictive text, saving time for a user of this device.

Bluetooth™ technology would also be present in Apple iPhone. It will empower a user to connect to other Bluetooth™ enabled devices with the aid of this mobile phone. However, the most prolific feature of Apple iPhone is the inclusion of various sensors to aid a user in clicking images, saving power and to enrich the user experience. As soon as a user picks up the phone and takes it near ears, one of its sensors detects the proximity of the device to the ear and switches off the display thus saving power consumption.

Synchronization of data with your PC or other devices too would be possible with the help of this device. Making calls, voicemails, SMS and clicking pictures and video clips would be possible with this futuristic mobile phone, Apple iPhone. Order and buying this mobile phone would be easy as Mobile Phone Shop UK accepts online booking of this communication device. Book your order now and have it in your hand when it is released in the market.

Nokia N75

2007-05-16T08:42:00.000-07:00

Service Provider: AT&T Wireless, Cingular Wireless

Operating System: Symbian OS

Screen Size: 2.4 inches

Screen Details: 2.4", 320x240 TFT main screen; 1.4", 160x128 front screen

Camera: Yes

Megapixels: 2

Bluetooth: Yes

Web Browser: Yes

Network: GSM, UMTS

Bands: 850, 900, 1800, 1900

High-Speed Data: GPRS, EDGE, UMTS

Special Features: Music

Nokia's first US 3G phone, in fact Nokia's first US flip smart phone, and AT&T's first 3G Symbian OS phone, the glamorous Nokia N75 has been the subject of many hopes and dreams in the seven months since it was first announced. It's beautiful and powerful to be sure, but poor battery life makes me worry you won't be able to get the use you want out of it.

MOTORAZR2V8 phone. Nice phone

2007-05-16T08:16:00.000-07:00

MOTORAZR² V8 brings the revolutionary RAZR form to a new level of sophistication. It takes the impossibly sleek RAZR profile and makes it even slimmer. It packs a powerhouse of features into a strong stainless steel structure. All told, the phone that has forever captivated is now about to mesmerize.

Certain mobile phone features are dependent on the capabilities and settings of your service provider’s network. Additionally, certain features may not be activated by your service provider, and/or their network settings may limit the feature’s functionality. Always contact your service provider about feature availability and functionality. All features, functionality and other product specifications are based upon the latest available information and are believed to be accurate; however such product specifications are subject to change without notice or obligation.

Supports the following Bluetooth profiles: A2DP, AVRCP, GAVDP, generic access, hands-free, headset, dial-up networking, object push, file transfer, generic object exchange, basic imaging (sending/receiving images supported only), basic printing (in the following applications: Phonebook, Datebook, Picture Viewer and Messaging).

About traditional vietnames costume

2007-05-09T09:08:00.001-07:00

AO TU THAN

The "ao dai" ("flowing tunic") has been the traditional dress for Vietnamese women long, long ago. There are many different kinds of ao dai: the four-part flowing tunic had two equal front flaps that women tied together, while the five-part flowing tunic had an additional small front flap that buttoned up onto the right side of the dress.

Different regions of the country have their own styles of flowing tunic. In the north, Vietnamese women usually wear the four-part flowing tunic, refers to as "Ao Tu Than", with a long skirt. The hat is called "Non Quai Thao".

On the right, instead of wearing "non quai thao", Nothern women just simply wear a scarf as a variety.

Student Uniform

In the Central Region, and in the South of Vietnam, "ao dai" is worn. It is a long flowing tunic that consists of only 2 parts and is worn over a loose-fitting pair of either white, black, or colored pants. However, the white pants are usually preferred over the others. The dress splits into a front and back panel from the waist down. It is also buttoned down on the 1eft side to the waist. There are many stylish variations in color and collar design.

White, or violet ao dai is also worn by Vietnamese students as their uniforms.

The cone-shaped hat is called "non la". Non la in these two pictures is more for posing and decorating. In reality, non la is more functional than decorative. It is used indeed like a hat to protect one from the heat and the sun in everyday life situation. However, in weddings (brides), festivals, or any formal event, a headdress (a diadem - as in the middle picture) is usually preferred. Note: neither non la, nor the headdress is required to be worn with ao dai.

Ao Dai with headdress

Ao Dai with Non La

Men Attire

Anyone who has seen the exquisite costumes worn by Vietnamese women will recognize similarities in the traditional dress of the male. Men costumes are worn with the conventional snug collar and buttoned down on the 1eft side to the waist, with no crease in front or back. The male dress extends only to the knees and is more loose-fitting.

There are many variations on the basic theme. At the top of the list is the elaborate dress of the emperor and the mandarins. Their rank was shown in the display of color in the brocade and embroideries. Gold brocade with embroidered dragons was for the emperor only. Gold is the national color and the dragon heads the fabulous mythical animal world. Purple is the color reserved for high-ranking court mandarins, while blue is for those of lower rank.

Costumes worn for religious ceremonies also have their special colors. Dresses for ceremonial occasions usually have very wide and ample sleeves. Wedding dresses are similar to the popular fashions, and the color is usually purple or blue brocade. Dresses for mourning have frayed fringes or a line up the back and may be either black or white in color.

Definition of "TET" holiday

2007-02-19T02:31:00.000-08:00

Read it to find out more about "TET" holiday

http://en.wikipedia.org/wiki/T%E1%BA%BFt

Component in "TET" holiday

2007-02-19T02:09:00.000-08:00

Cay neu is a bamboo pole (New Year's Tree) stripped of its leaves except for a tuft on top. Red paper is used to decorate the tree, which is planted outside the house during the Tet holiday. The cay neu has Taoist origins and holds talismanic objects that clang in the breeze to attract good spirits and repel evil ones. On the very top, they frequently place a paper symbol of yin and yang, the two principal forces of the universe. Sometimes a colorful paper carp flag will fly from the top. The carp (or sometimes a horse) is the vehicle on which the Kitchen God travels to make his report. This tree is more common in the countryside now than in the city. It is ceremonially removed after the seventh day of Tet.

Vietnamese Peach Blossoms (Hoa Dao)

According to Vietnamese legend, once upon a time, in the East of the Soc Son Mountain, North Vietnam, existed a gigantic peach tree. The tree was so huge that its shadow extended through out a large area of land. Up on the tree, lived two powerful deities, Tra and Uat Luy. They protected the people of the land in the surrounding areas from the devils. The devils were so afraid of these two deities that even the sight of the peach tree haunted them.

However, at the end of every lunar year, these two deities had to fly back to heaven for an annual meeting with the Jade Emperor. During this time, the devils took advantage of this opportunity to harass the peaceful inhabitants. To fight the battle against these devils, people came up with the ideas of display a branch of the Peach tree in the house to scare away the devils. Since then it becomes a custom of the North Vietnamese to have a branch of a Peach tree during Tet season to protect themselves against the Satan soldiers. Those who don't have Peach tree can draw the figures of the two deities, Tra and Uat Luy, on red paper, and display them in front of the house.

Hoa Mai

While Peach tree is preferred in the North, Hoa Mai is more commonly used for this ceremony in the South because of the warm weather. Hoa Mai is a small, yellow flowering plant that is used for decoration during Tet with the meanings of prosperity and well-being for the family. The value of these flowers is determined by the number of petals - the more petals, the more expensive the flower.

Kumquat Trees

Kumquat trees about two or three feet tall are carefully selected and prominently displayed during Tet. To carefully choose a kumquat bush, the buyer must pay attention to the symmetrical shape, to the leaves and to the color and shape of the fruit. The bushes have been precisely pruned to display ripe deep orange fruits with smooth clear thin skin shining like little suns or gold coins on the first day of the lunar new year. Other fruits must still be green to ripen later. This represents the wish that wealth will come to you now and in the future. The leaves must be thick and dark green with some light green sprouts. The fruits represent the grandparents, the flowers represent parents, the buds represent children and the light green leaves represent grandchildren. The tree thus symbolizes many generations. Guests will caress the light green leaves about to sprout and compliment the discerning host who chose so carefully.

The "Mam Ngu Qua"

The "five-fruit tray" on the ancestral altar during the Tet holiday symbolizes the admiration and gratitude of the Vietnamese to Heaven and Earth and their ancestors, and demonstrates their aspiration for a life of plenty. As one theory goes, the five fruits are symbolic of the five basic elements of oriental philosophy: metal, wood, water, fire, and earth. Some people believe that the five fruits are symbols of the five fingers of a man's hand that is used to produce physical wealth for his own use and to make offerings to his ancestors. However, in a simpler way, the five fruits represent the quintessence that Heaven and Earth bless humans. This is one of the general perceptions of life of the Vietnamese, which is "When taking fruit, you should think of the grower". Today, the tray may contain five or more fruits, in the form of a pyramid like before or in an different shape. Regardless, it is still called the Mam Ngu Qua, the five-fruit tray.

Fire Crackers

The most exciting element in the celebration of Tet is the lighting of trang phao (fire crackers). These explosions are believed to drive off ghosts and evil spirits and leave good luck in their place. As thousands of households simultaneously partake in this fantastic part of Tet, the level of volume and excitement rises to a fury. This level of emotion is the most memorable part of Tet and also the part which makes it such a marvelous experience. However, firecrackers are no longer used as the government banned them in 1995.

Banh Chung

It is a square cake, wrapped in banana leaves and tied with laces of flexible bamboo slivers. It is a very rich food for the interior contains a filling of bean paste to which may be added small bits of pork meat, both fat and lean. This filling, which is amply seasoned, is pressed between layers of glutinous rice. Its square shape is considered a symbol of the thankfulness of the Vietnamese people for the great abundance of the Earth, which has supplied them with nutritious food throughout the four seasons of the year.

Cau Doi (Parallels)

Composing, challenging and displaying parallels represents an elegant cultural activity of the Vietnamese. On the occasion of Tet, parallels are written on red paper and hung on both sides of the gate, the pillars or the ancestral altar. Each pair of parallels has an equal number of words with contrasting or corresponding meanings and lines of verses. They show a keen intelligence, perception of nature and social life, uphold morality and a yearning for the well-being of all people. The red is symbolic of auspicious and powerful vitality, according to popular belief. Mingling with the green of the banh chung, the pink of the peach blooms, the yellow of the hoa mai, and the red of the parallels is sure to make the Spring warmer and cozier.

Xin Xam

After Giao Thua, the last day of the lunar calendar year, many Buddhists go to their favorite pagoda to pray for a good year and to get a fortune reading for the whole year. Each person will get a chance to shake the tube that contains reading sticks, until one stick falls out of the tube. In the case that many sticks drop out of the tube, that person will have to repeat the process. Afterwards, usually the monks will translate the meanings of the reading.

2007-02-19T01:58:00.000-08:00

"Nguyen Hue" Street , the beautiful street in viet nam, in the January 1st , lunar year

Picture of the "TET" holiday in viet nam

HAPPY VIET NAM NEW YEAR

2007-02-19T01:44:00.000-08:00

Happy new year to you, my family, my dear friend. We call this day is "Tet" holiday, and i want to tell all of you about "TET" holiday in viet nam.
WELCOME YOU TO VIET NAM

Before "TET" holiday

Some days before Tet Ong Tao (on the 23rd of the twelfth moon), people start their preparations for the Kitchen God's journey to the Heaven to make his report to the Jade Emperor. This report includes the year's activities of the household in which he has lived. A farewell and thank-you dinner is given to the Kitchen God at Tet Ong Tao. The Kitchen God will need a week for his mission to Heaven.

After the Kitchen God has gone to the Heaven, preparations for the New Year festivities begin in enthusiasm. The week before New Year's Eve is called a period of "Tat Nien". Tat Nien (literally meaning the end or "to extinguish the year") is the celebration of the last session of a period, such as the last class of school, the last day in the office, even the last bath, all with parties and great ceremonies.

Some families set up a Tet tree outside the house in the week before New Year's Eve. The Tet tree, called cay neu, is a bamboo pole stripped of its leaves except for a tuft on top. It is supposed to ward off the evil spirits during absence of the Spirit of the Kitchen God.

Sweeping and scrubbing must be done during this time as tradition discourages it during the Tet holiday. Two items required for the proper enjoyment of Tet are peach flower branches and kumquat trees. Throughout the country, on bicycles of roving vendors, flowers create great splashes of color. In the north, the soft rose-colored dao peach flowers decorate homes and offices while the bright golden yellow branches of the hoa mai are preferred in the south.

Kumquat trees, about two or three feet tall, are carefully selected and prominently displayed. To carefully choose a kumquat tree, the buyer must pay attention to the symmetrical shape, to the leaves and to the color and shape of the fruit. The bushes have been precisely pruned to display ripe deep orange fruits with smooth clear thin skin shining like little suns or gold coins on the first day. Other fruits must still be green to ripen later. This represents the wish that wealth will come to you now and in the future.

When Tet is approaching, crowds of shoppers at the markets become thicker and more frantic each night, holding up traffic as they jostle each other to reach the counters with the best buys. Prices are a bit higher, but thriftiness is not considered a virtue at Tet.

While shoppers roam the streets, banh chung patties wrapped in leaves are steaming in giant vats. After being boiled until the outside of banh chung has taken on a lovely light green tinge, it is taken out of the vats and cooled. Banh chung will be eaten and used as offerings to worship ancestors during Tet.

Before the New Year's Eve, shops, stalls and restaurants are locked, leaving a notice hung on the door announcing the date of reopening. Special dishes must be completed to serve the family and its guests for the first three days of the new year.

Hành trình viết ứng dụng corba đầu tiên

2007-02-19T00:24:00.000-08:00

The first word i want to tell you is that my english is terrible. So please correct me if you find out incorrect sentences.
First, i want to tell you why i have to write the distributed application using CORBA technology. It is because of my teacher. He said: All you guys have to write an project about any topic you want but ( always "but" ) ... about linux world . Omg, what topic i have to write now ??? Many questions and ideas appear in my mind in respectively but it seems to be difficult and i have no time. :( . (Lazy man ^_^ ) . One day, i found the world CORBA ( actually i saw it when i was at university but can not understand what it means). And i chose CORBA for my project.

So now, the first thing: what the hell it is ???
Like other students, i search it in internet , using google.com, keyword : corba
Oh no, many results from google. I hate that. Maybe, i have to chose another keyword. I think: searching is art and the searcher is artish ^^.

I found this definition in http://en.wikipedia.org/wiki/CORBA
The Common Object Request Broker Architecture (CORBA) is a standard defined by the Object Management Group (OMG) that enables software components written in multiple computer languages and running on multiple computers to interoperate.

It is so bad, i only understand only above sentences among many sentences, and do not understand any others sentences. I'm really a stupid man( or maybe , that website is stupid, i don't know).
But this is enough for the question : "what is CORBA about?"

The second things to do: So what package i have to install to use CORBA. Next, i have to use google again. Thans google.com
The next keyword is linux corba package . I found the website
http://linas.org/linux/corba.htm
My thanks to this website, i found the omniORB is the package i want to install( support C/C++, platform: linux and windows, fast and free )

So now i download and install it in my linux host.
(To be continue...)